Are there some first off opinions about how to manage Media devices?

[dcorsus]

This is not a thread to debate whether we need the MediaAPI or not but just to see what ideas or guidance is living out there.

In HS2 we had a musicAPI which was pretty much hidden from any HS control screens and just a way to communicate between PIs and HS Touch enable devices. The API was limited and rigid and had pre-baked in limited ways to navigate content, but only music content. The good thing was it didn't clutter any device management pages AND it allowed a user friendly solution for those who were creating HSTouch screens.

In HS3 we had a MediaAPI which had all the right intentions, and I'm sure opinions will differ here, but couldn't get enough support from the HS team to really make this live up to its potential. In my opinion, it appeared that when it was actively being working on, it was pulled towards making the MediaAPI work like the musicAPI, so that (by now long abandoned) PIs such as iTunes and MediaPlayer would work in HS3. I think most of us developers who implemented the MediaAPI got most of it working, but for me, it needed a lot more TLC to make it a robust interface. I never got the "new navigation" mechanisms to work for other content then music and the biggest concern was that it was chatty like crazy. I could see HS Touch screens, over and over, requesting large parts of the track database creating very unresponsive HST screens. So let's put that aside ...

In HS3, we had a new way to create parent/child relationships and we had device types that could (at some point in the future) be used to create HST screen without using the MediaAPI. So I ended up creating something like 20-ish devices PER PLAYER. Things like player control, track info, next track, player position, Art, next track Art ......

So here is where hopefully this discussion starts, what do we do different for HS4?

In my opinion, my HS3 implementation for the Sonos PI was bad but I followed the guidance that was available then and I've seen some users create amazing HST implementation (albeit some of them ended up scripting the heck out of it all). I have 18 or so players so I have a few hundred devices that the PI created ..... just to be used for creating HST screens. Guess what ..... I stopped believing in managing a complex device like a Sonos player from an HST screen so they just clutter my device management pages and I'm sure it did for many reading this. So I actually would not like to repeat this, if I want to do better in HS4.

I like to simplify my thinking, so I see 2 major components to controlling media:

• Player control
  • This is things like, pause/play/stop/next/prev/Volume/position/track info/position info/art ....
• Content control
  • This is navigating content, starting with any particular root and then navigate down or up, select ....

1. Player Control
It is my recommendation that these controls stay as HS device controls but all attached to the same root/parent, it could all be exact like it was provided in HS3 (the DeviceTypeInfo class). It is my recommendation that we revisit the class and add eTypes that we believe are missing. Then it is my second recommendation that the HS team provides a more optimized "device view" for a media device. See the example below turn the current device layout a bit into a mini player layout. This could be a defined layout by the HS team or allow us developers to customize it.


2. Content control
This part tends to have a few pieces to it. Some sort of a pane where you start navigating up/down, select etc and some other pane which tends to represent your selection (or queue). The latter when used to mimic the content of the player's queue will show what is currently playing, allow you to select other items etc.
It is my recommendation that these functions will NOT be represented as HS Devices with tons of children. I think the solution here is a substitute or a completion of something like our MediaAPI. Given that navigating content or representing a queue is in essence a Listbox. I would further recommended that the HS team focuses on creating an enhanced listbox HS device and how to best show it on the future UIs. Let us PI developers manage the content! I would NOT store the content of the listbox in the HS database (to prevent you store 100,000 track items) but use the same mechanism you have today to inform a PI that a device was activated. I could see events such as "load content", "single click", "double click", "selection". There is a lot more to it, but just wanted to describe the high-level here as a proposal. As an FYI, I believe a listbox like this can be used for non-media PI developers to show for example list like, action items, email, log items that require action etc.

So what are other developers thinking and more importantly, how does the HS team see the future of managing and automating Media devices around the house?

Dirk

[rjh]

I would like to move this along and propose a possible solution. I am not sure I follow your idea of not using parent/child devices. But let me run this by you.

We would create some new device types, similar to how the thermostat works. There would be a high level device type of "MEDIA" set on the root device, and then sub types like:

Artists
Albums
Playlists
Tracks
Genre
Play
Pause
Stop
Forward
Back
Album Art

And whatever else is needed

With this information we can now create a UI for any plugin that has media control. The plugin author will not need deal with the UI.

We still use the parent child as the children are just "features" of a device. So you would have a root device for each player and then child devices for all the controls and status.

This content is dynamic, if a user selects an artist, you get the control selection and then load the other controls/droplists as needed, like albums,tracks, artwork, etc.

We provide a flag on the device that you would set on a dynamic child named "DYNAMIC_CONTENT". If this flag is set, it tells the UI that this device and its children can change when this device is controlled. The UI then waits for a change/set on this device. After you load the content (update any child devices), updated the value of the selected device, the UI will then refresh displaying the new content.

Some possible issues. You may want more control of the layout of the UI and for this we could honor the row/col settings on the controls so you can place the controls where you want them. If you want even more control, we will have to discuss on how to accomplish this.

Another issue is data size. If someone selects a genre and the user has 50000 tracks, how quickly can a droplist be loaded? Might make sense to load some fixed max, like 500 items, then a "Load More" selection at the end to load the next batch.

We have to also make sure this works over the JSON interface, which I think it will. Callers will know that the content is dynamic based on the flag and can call back to get the updated device status/controls. HS would not return to them when a dynamic device is controlled until all the child devices are updated.

How does this sound? Am I missing anything?

Could have this implemented almost immediately.

Note that we are looking at some speed improvements to the plugin communications API, many thanks to bsobel on this, which will should allow things to work much better over the old media API.

[dcorsus]

I would like to move this along and propose a possible solution. I am not sure I follow your idea of not using parent/child devices.

Actually I DO want to use parent/child devices, must have not been clear. We might need to poll to add MORE child device types.
What I didn't like is that all these devices showed up in my management screens, which was way too much of a clutter. I probably could have set them to hidden but seem to recall some issues with that early on in HS3.

We would create some new device types, similar to how the thermostat works. There would be a high level device type of "MEDIA" set on the root device, and then sub types like:

Artists
Albums
Playlists
Tracks
Genre
Play
Pause
Stop
Forward
Back
Album Art

And whatever else is needed

With this information we can now create a UI for any plugin that has media control. The plugin author will not need deal with the UI.

We still use the parent child as the children are just "features" of a device. So you would have a root device for each player and then child devices for all the controls and status.

YES. We actually have all of this, it is already in my HS3 implementation, we just never used it anywhere. So if you could provide a layout customized for media playing (see my picture above) .... a bit like you do for thermostats, that might already be a big step forward.

This content is dynamic, if a user selects an artist, you get the control selection and then load the other controls/droplists as needed, like albums,tracks, artwork, etc.

We provide a flag on the device that you would set on a dynamic child named "DYNAMIC_CONTENT". If this flag is set, it tells the UI that this device and its children can change when this device is controlled. The UI then waits for a change/set on this device. After you load the content (update any child devices), updated the value of the selected device, the UI will then refresh displaying the new content.
.

I'm glad you brought this up as I was thinking about it ( :-) ) . I wasn't thinking about dynamic content for track info but for example, current track position info, is totally dynamic and I already spotted "flickering" in the HS4 device screen, when my PI updates the track position every second. I also noted that status when selecting the children wasn't being updated.

I'd be curious to hear other people's views whether ANYTHING needs to be persistent wrt to media information

This content is dynamic, if a user selects an artist, you get the control selection and then load the other controls/droplists as needed, like albums,tracks, artwork, etc.

We provide a flag on the device that you would set on a dynamic child named "DYNAMIC_CONTENT". If this flag is set, it tells the UI that this device and its children can change when this device is controlled. The UI then waits for a change/set on this device. After you load the content (update any child devices), updated the value of the selected device, the UI will then refresh displaying the new content.

I would say "yes" unless I'm missing something here. Would this be different from what it is today? What I mean is, if you add a "dynamic_content" flag on the root/child devices, I would think you don't have to do DB updates for those devices and leave it up to the PIs to make sure the content is set properly at start-up and maintained properly during course of its execution. I do want to iterate my view of PLAYER control versus CONTENT SELECTION. I'm here arguing about PLAYER control. Content selection absolutely is dynamic but I'll argue that below.

This content is dynamic, if a user selects an artist, you get the control selection and then load the other controls/droplists as needed, like albums,tracks, artwork, etc.

We provide a flag on the device that you would set on a dynamic child named "DYNAMIC_CONTENT". If this flag is set, it tells the UI that this device and its children can change when this device is controlled. The UI then waits for a change/set on this device. After you load the content (update any child devices), updated the value of the selected device, the UI will then refresh displaying the new content.

Some possible issues. You may want more control of the layout of the UI and for this we could honor the row/col settings on the controls so you can place the controls where you want them. If you want even more control, we will have to discuss on how to accomplish this.

Another issue is data size. If someone selects a genre and the user has 50000 tracks, how quickly can a droplist be loaded? Might make sense to load some fixed max, like 500 items, then a "Load More" selection at the end to load the next batch.

We have to also make sure this works over the JSON interface, which I think it will. Callers will know that the content is dynamic based on the flag and can call back to get the updated device status/controls. HS would not return to them when a dynamic device is controlled until all the child devices are updated.

How does this sound? Am I missing anything?

So now let's talk about Content selection. I envision a new "root" type ala Media but say called "Content or listobject". I see this further linked to child devices that can be created and attached to that specific root. These would obviously at the decision of the PI author but it could be control buttons to do things like "level up" or buttons that will load say tracks or albums or action items. Maybe also here you can suggest a enum of predefined functions that could "automate" UI screens. Not sure I'm making myself clear here, so here's an example what I would do ....

a/ create this imaginary listbox as root device. The PI can add/delete etc items to this, preferably in some "HTML" fashion so we could potentially populate with album art thumbnails. HS would provide events (like it does today through SetIOxx), say click on item, load, selection .... It also allows to high-light items (important when used as queue)
b/ I would create child devices with Label UP, Artist, Album, Track, RadioStation, Video .... They act like regular buttons and I get SetIO notifications when clicked so I can update the content of the root listbox
c/ a new child type, like a html string with hyperlinks in it. Think of it showing your navigation path. It may show something like 10CC->Greatest Hits->I'm not in Love allowing a user to quickly navigate back OR show how he/she got to the content shown in the list box.


See an example of how my player and control control page is laid out today

[alexbk66]

rjh
This might work, but what you are proposing is only for media devices. It's very limiting. If somebody needs to add some new type of control - you will have to update your code for that?

And for other types of devices you will have to re-invent the wheel again? Theese days there's a lot of new technologies emerging, not just thermostats or media players.

The software should be generic enouh - so you don't have to do anything special for each device type. I know, it's a tough task - but thtat's what you are paid for

[dcorsus]

Forgot to mention, to avoid that we need to "transmit" thousand and thousand of objects like tracks, perhaps the listbox object can generate or provide information of what is visible and how much. What I mean by that is we either make it a responsibility of the PI authors to populate objects in the listbox that are visible at that point AND react fast enough (Just In Time JIT) to provide new content when the user scrolls. Alternative implementation is that this is done by the HS4 framework, only to show JIT information and optimize communication. Having that said, HS might still end up with large memory needs if say you have 32 Sonos players (which at least one user has) times say 50,000 tracks, that would be a LOT of memory.

We could also turn this on its head, rather than creating a player and content navigation solution for EACH device, you just create one per say HSTouch device or just one for the PI but first selection criteria is selecting which device you want to control. People who have been building HS Touch screens, I believed like this a lot better over creating n player (similar) screens, with one dedicated screen for each player because there was no way in HS3 to dynamically pick up the correct HS references. At least one of my users, scripted the bejeezus out of his implementation and on the same HS Touch screen he can select a player and all buttons, text boxes and art-work are dynamically pulled from HS with the correct HS references.

Opinions?

[dcorsus]

rjh
This might work, but what you are proposing is only for media devices. It's very limiting. If somebody needs to add some new type of control - you will have to update your code for that?

And for other types of devices you will have to re-invent the wheel again? These days there's a lot of new technologies emerging, not just thermostats or media players.

The software should be generic enough - so you don't have to do anything special for each device type. I know, it's a tough task - but thtat's what you are paid for

Totally agree. The suggestion was to "standardize" the basic stuff so it can be easily be automated, say linked to a pre-defined layout BUT more complex stuff would have to be dealt with through own created GUI screens. That way, the novice users gets what he/she want easily and the expert can do things to his/her heart content. In fact, today, in HS Touch you could create/save and share templates. However, for HS3, for media, that function was never finished, it did work in HS2 with the MusicAPI though and was actually used by some of my users. I see no reason why this couldn't happen in HS4 and I think Rich or someone has already indicated you could add this to the html folder (at some point :-)

Dirk

[sirmeili]

rjh
This might work, but what you are proposing is only for media devices. It's very limiting. If somebody needs to add some new type of control - you will have to update your code for that?

And for other types of devices you will have to re-invent the wheel again? Theese days there's a lot of new technologies emerging, not just thermostats or media players.

The software should be generic enouh - so you don't have to do anything special for each device type. I know, it's a tough task - but thtat's what you are paid for

They should always look for the "generic" approach and apply that. Make it flexible enough to work for anything. Allow users/developers to define "tile groups" if you want to call them that. It allows them to define the layout and controls that appear from a group of devices on a "tile" (sine the UI is very tile based).

To begin this can be completely text configurable for the plugin devs to do. I know I'm not scared to do it. Require it in XML, JSON whatever you want. Hell, if you did XML, you could just use XSLT to transform it to HTML for you if you defined the contract properly.

[rjh]

In regards to device types, yes we have some, I was concerned that if we use them that the UI might be messed up for HS3 plugins. But if you think thats ok we can use them. What we have then is:

Public Enum eDeviceAPI
                No_API = 0
                Plug_In = 4
                Security = 8
                Thermostat = 16
[B]                Media = 32[/B]
                SourceSwitch = 64
                Script = 128
                Energy = 256
            End Enum

            Public Enum eDeviceType_Media
                Player_Status = 1
                Player_Status_Additional = 2
                Player_Control = 3
                Player_Volume = 4
                Player_Shuffle = 5
                Player_Repeat = 6
                Media_Genre = 7
                Media_Album = 8
                Media_Artist = 9
                Media_Track = 10
                Media_Playlist = 11
                Media_Type = 12
                Media_Selector_Control = 20
                Root = 99
            End Enum

For the dynamic content, yes the UI will update now if a device changes, but only that device. I am proposing that if you set the DYNAMIC flag that the UI will update all of the devices that are under the root device referenced by the child that is changing. Most likely changing an artist will change the artwork, the player status, etc. So we will just refresh everything.

I think with this we can build a UI with the child controls. I would rather not create more custom controls just for media. You could add a "back" button for navigation if you feel that is required, and maybe a label to show where they are. Some possible issues is that under some views you might want to hide some controls. For example when the view is first displayed you only display the genre's droplist, then other droplists appear as they make selections. You can either delete and added child devices (might be hard to manage), or we add another "hidden" attribute that you set, or you remove the media attribute (or we create a media attribute that says to hide it for now). We never really had a way to show a control but have it be disabled, so maybe a "disabled" attribute is needed so you can show controls but have them be disabled.

You can of course create a feature page with a custom player on it, but it would be ideal if that was not necessary.

I can put together a test UI with this and we can see how well it looks and works.

I like this because it does not require any changes to the JSON interface or our value pairs.

[dcorsus]

In regards to device types, yes we have some, I was concerned that if we use them that the UI might be messed up for HS3 plugins. But if you think thats ok we can use them. What we have then is:

Public Enum eDeviceAPI
No_API = 0
Plug_In = 4
Security = 8
Thermostat = 16
[B] Media = 32[/B]
SourceSwitch = 64
Script = 128
Energy = 256
End Enum

Public Enum eDeviceType_Media
Player_Status = 1
Player_Status_Additional = 2
Player_Control = 3
Player_Volume = 4
Player_Shuffle = 5
Player_Repeat = 6
Media_Genre = 7
Media_Album = 8
Media_Artist = 9
Media_Track = 10
Media_Playlist = 11
Media_Type = 12
Media_Selector_Control = 20
Root = 99
End Enum

I hope others like spud and pcp are reading this, perhaps others who have media type of PIs, they may want to chime in. I can confirm that I followed the proper enumeration in HS3. I can post an HS4 database if you like where my Sonos PI created the devices, just let me know.

I would rather not create more custom controls just for media. You could add a "back" button for navigation if you feel that is required, and maybe a label to show where they are. Some possible issues is that under some views you might want to hide some controls. For example when the view is first displayed you only display the genre's droplist, then other droplists appear as they make selections. You can either delete and added child devices (might be hard to manage), or we add another "hidden" attribute that you set, or you remove the media attribute (or we create a media attribute that says to hide it for now). We never really had a way to show a control but have it be disabled, so maybe a "disabled" attribute is needed so you can show controls but have them be disabled.

I personally was hoping for a listbox and not a droplist. The latter is (in my opinion) not very user friendly and I don't believe you can do a "selection" of items. Content is typically navigated in list boxes or tables as opposed to droplists. Today in HS3 I use droplists in my event/action screens, allowing a user to select content. Its OK because you create events very sporadic as opposed to navigating content.

You can of course create a feature page with a custom player on it, but it would be ideal if that was not necessary.

I think that is what most of us ended up doing. I think I wrote it somewhere, if you could find a good way to integrate that with HS Touch, I'm good. If say, a user can click on that "control" button that doesn't do anything today, and that would pop up a little overlay of sorts that represent our own created feature page with a client/server architecture we had in HS2 with ASPX pages or HS3 with JQuery components, that could potentially work. I do fear however, if the PI author ends up resposible to create this feature page and HS is promoting the new HS4 interface for mobile, it pretty much means that users will directly complain to PI authors about layout on some version of a mobile device. I'd rather see HS come up with a framework we can work with that looks good and resizes properly on any mobile device.

I can put together a test UI with this and we can see how well it looks and works.

Sure go ahead. I can mail you my HS4 database that has many media devices already in it.

I like this because it does not require any changes to the JSON interface or our value pairs.

Not sure what this mean but I believe you .....

[rjh]

I understand what you are saying about the list boxes, but on smaller screens a listbox might not work well. We use the material design select list from here:

https://mdbootstrap.com/docs/jquery/forms/select/

These display nicely on smaller devices. There is an option to do multiple selections (although we will need a way to specify this mode).

I will code something up and we can see how well it works.

[dcorsus]

I understand what you are saying about the list boxes, but on smaller screens a listbox might not work well. We use the material design select list from here:

https://mdbootstrap.com/docs/jquery/forms/select/

These display nicely on smaller devices. There is an option to do multiple selections (although we will need a way to specify this mode).

I will code something up and we can see how well it works.

Will need to get used to it I guess. Also realized that multiple selection as I'm used to with ctrl or alt click, actually don't exist on mobile devices do they. If I take the Sonos controller APP itself, it does have some sort of list box, but each line item actually has "embedded" drop down control, where you can select to add at start, end of queue, play next etc. So the option to embed images, text and perhaps other controls per list item would be great :-)!
An example will be a really good start.
Dirk

[rjh]

Dirk, it looks like you are using a custom device type for things like player controls (play, pause, etc.)? I propose we move those to the "CONTROL_USE" types. Reason being is that you then you can put all your player controls on a single device, as you are doing now. The device type is per device, not control. I could then create a device type of "MEDIA_CONTROL" so the UI knows that the child devices contains media controls. Does this make sense? We can still use device types for more complex controls like volume and balance.

[dcorsus]

Dirk, it looks like you are using a custom device type for things like player controls (play, pause, etc.)? I propose we move those to the "CONTROL_USE" types. Reason being is that you then you can put all your player controls on a single device, as you are doing now. The device type is per device, not control. I could then create a device type of "MEDIA_CONTROL" so the UI knows that the child devices contains media controls. Does this make sense? We can still use device types for more complex controls like volume and balance.

Here is what I do:

I have for each player a "root" device set to:

DT.Device_API = DeviceTypeInfo.eDeviceAPI.Media
DT.Device_Type = DeviceTypeInfo.eDeviceType_Media.Root
DT.Device_SubType_Description = "Sonos Player Master Control"

This device has a few buttons associated with it, such as play,stop, pause, prev, next, shuffle, repeat, vol-up, vol-dn, bal-left, bal-right, loudness
All of these are player control functions. Is this what you mean by custom device types?

Then I create children devices attached to the root player device. These are the devices that in my opinion, clutter my device view, but they were easy to drop into HS Touch screens or used to trigger events when track names are changing etc.


Each player creates 18 children. They are:

• Current track info
  • Track name (DeviceTypeInfo.eDeviceType_Media.Media_Track)
  • Artist Name (DeviceTypeInfo.eDeviceType_Media.Media_Artist)
  • Album Name (DeviceTypeInfo.eDeviceType_Media.Media_Album)
  • Album Art (image and URL)
• Next track info
  • Track name
  • Artist Name
  • Album Name
  • Album Art (image and URL)
• State (DeviceTypeInfo.eDeviceType_Media.Player_Status)
• Volume (eviceTypeInfo.eDeviceType_Media.Player_Volume)
• Mute
• Loudness
• Balance
• Track Length
• Track Position
• Radio Station Name
• Repeat
• Shuffle

Most of these are mapped to DT.Device_Type = DeviceTypeInfo.eDeviceType_Media.Player_Status_Additional because there is no proper enum available.

So to represent a player, if requires some parts of the player device (to control it) and some parts of the children devices, the latter which hold info on what is being played, settings etc.

So it now shows like this , which I think is a hot mess (on my behalf)



I think all the art URL info is not needed here but is critical to drop into HS Touch screen, linked to an image box. So how do we solve both, have some setting they don't show here but only show when HS Touch screens are created?

Secondly, how to we get to something like this?


Here is how it looked in HS3


Thoughts?


Dirk

[rjh]

We can get close to what you want, but I am trying to get there by providing a "standard" way of describing the controls so any media player can simply use what we have. There will also be some custom controls but I think we can handle most.

So I propose using the MEDIA device type for the root, then media sub types for the children, and a grouping sub type for the common controls where each control has a control_use assigned to it so the UI knows what the control is.

I have a media plugin built and am adding the controls. I will get that put together and we can see how well it works and maps to what you have.

I would not worry about HSTouch, I can get that fixed up later.

I see that in your HS3 device view you do not allow for selecting Artists's, etc. I think it would be nice to have that in the new UI.

[dcorsus]

I am trying to get there by providing a "standard" way of describing the controls so any media player can simply use what we have.

AMEN! I thought I had done that for HS3 but your example when ready will show me what I forgot doing.

I see that in your HS3 device view you do not allow for selecting Artists's, etc. I think it would be nice to have that in the new UI.

I didn't (or didn't know) how to properly implement things like selecting Artists IN THE DEVICE view. I did implement it using jQuery in customized webpages linked to the PI OR in the event/action screens when a user wants to say, create an action to play track x from Album y from Artist z.
Here is a view of the player custom page I had for HS3


or how you would create Sonos Actions:



The brain freeze I still have is how I would have done this in HS3 in the devices view. That's why I was suggesting some "enhanced Listbox" or as you suggested "drop list" that could wrap around all things navigation. It is my view, that in order to select items from a navigation tree, without the ability to easily show the navigation path, it really is not a great user experience. Say you navigated down to a track called "happy song", there could be tons of tracks with the same name from different artists or from same artist but different albums. So it is my view that if we want to add navigation, it needs to be user friendly, including some view how we got where we are.

Dirk