Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ITV Hub is now ITVX #428

Open
gacres2 opened this issue Nov 17, 2022 · 66 comments
Open

ITV Hub is now ITVX #428

gacres2 opened this issue Nov 17, 2022 · 66 comments

Comments

@gacres2
Copy link

gacres2 commented Nov 17, 2022

The ITV Hub is no more - and it is now ITVX with its subscription service overlaid on the old hub.

As a result, searching for ITV programs fails for me as of today - and "Use Current Page" from ITVX does not work. GiA finds 12000 programs but stalls at that point.

Turning off ITV programs in Preferences got me functionality back for iPlayer programs, but it looks like the ITV side of the house will need some repairs.

@nrj53
Copy link

nrj53 commented Nov 17, 2022

I agree. Neither of the last two releases work for ITV anymore. It caches 22,817 programs but fails to start the ITV cache process. As a result the program then hangs indefinitely. If you turn of ITV in the preferences, the program works normally for iPlayer programs. This just started today, since it was working normally 12 hours ago.

@ejstubbs
Copy link

I am seeing this as well. The hang is disconcerting since quitting GiA triggers a warning that series link info may be lost, which would create awkwardness should it happen.

@pebceb
Copy link

pebceb commented Nov 17, 2022

Same problem for me. BBC shows still okay if I turn off ITV like suggested. Hopefully Scott will have a fix.

@skovatch
Copy link

The short fix is to turn off ITV in the preferences, but I will release an update that disables ITV caching until I can figure out if ITV will ever work again.

@pebceb
Copy link

pebceb commented Nov 17, 2022

Just one question Scott. Turning off ITV allows us to get BBC shows but we still cannot use the current web page option for ITV shows. Is there another workaround to get them?

@skovatch
Copy link

Command-line yt-dlp won't download from the new ITV, and even Downie can't get a stream to save either, so there's no way to download ITV content at all at this point.

More concerning to me is that Downie can identify that there is a video stream but then fails claiming the video is DRM-protected. If true, that's the end of ITV support.

STV is still a possibility, though.

@dvd403
Copy link

dvd403 commented Nov 18, 2022

I tried STV but it insists I sign in.

So I created an account even giving an accurate but false postcode in the area served by STV to be able to sign up. It says success! but when I try to stream it says I am outside the area.

I am using a UK VPN - I think it's still detecting I am not in the UK - I am in the middle of the Pacific Ocean - but it may be, like 4OD does, also checking the time zone on the computer to see if it matches the postcode. I'll have to try with my other mac set to UK time zone as soon as I finish repairing it.

@pebceb
Copy link

pebceb commented Nov 18, 2022

I got STV to work in my browser with a VPN. Not a time issue - you must have set up a postcode in Scotland. I picked Edinburgh.

@raythw
Copy link

raythw commented Nov 18, 2022

The very latest version of Downie seems to have no problem with downloading from the itvx site. However, you'll get an 'Error: Cannot find download ID' message if it's a show that requires a premium account. I don't know what happens if you have such an account and use Downie. I suspect DRM would be in use for shows that require such a premium account. The problem in GiPA seems more to do with caching than actually downloading...

@hugofalkman
Copy link

Submitted a ticket for ITV yesterday to Charlie, the creator of Downie. This morning European time he responded: " Hi, thanks for the report. This issue will be fixed in the next update. Cheers, Charlie Monroe [[email protected]]" This afternoon CET, release 4.6.1 shipped fixing ITV. Release notes:
"Downie 4.6.1 Nov 18, 2022
New: floracing.com, gamer.tw.
New: Custom integrations now support post-load delay for execution.
New: Custom integrations now support description metadata field.
New: Custom integrations now allow passing a context dictionary between themselves.
Improved: arte.tv, bilibili.com, crunchyroll.com, douyin.com, itv.com, iqiyi.com, omny.com, reddit.com, soundcloud.com, uktvplay.co.uk, youtube.com.
Improved: Live streams that are downloaded as TS containers will now convert to MP4 when postprocessing is set to MP4."

@raythw Submit a new ticket for your error message -- Charlie willl tell you if its DRM, or can be fixed.

@m1ctk
Copy link

m1ctk commented Nov 18, 2022

I thought it was the launch of ITVX messing up things. Sao that's the end of automater I guess.

@skovatch
Copy link

I thought it was the launch of ITVX messing up things. Sao that's the end of automater I guess.

No, GiA will continue on. It may take some time but scanning ITVX for show listings should still be possible. And if Downie can download ITV content I'm pretty confident yt-dlp or youtube-dl can do it too.

@skovatch
Copy link

You may want to follow along with this yt-dlp issue tracking ITVX support. It's not encouraging, honestly. yt-dlp/yt-dlp#5570

@ejstubbs
Copy link

Living in Edinburgh and already having an STV Player account, I can access programmes on there just fine. But GiA doesn't recognise them when I click the "Use Current Webpage" button, saying:

Programme Page Not Found

Please ensure the frontmost browser tab is open to an
iPlayer episode page or ITV Hub episode page.

Did this ever work, or would it need a change to GiA?

(I'm particularly interested in this because I've just discovered that STV Player currently has The Field of Blood available, something I've been trying to get hold of for ages!)

@skovatch
Copy link

skovatch commented Nov 18, 2022

Programme Page Not Found
Please ensure the frontmost browser tab is open to an
iPlayer episode page or ITV Hub episode page.

Did this ever work, or would it need a change to GiA?

No, although it has been requested in the past. See #420

@ejstubbs
Copy link

OK, thanks. I shall await developments.

@pebceb
Copy link

pebceb commented Nov 22, 2022

Any update on a possible fix for the ITV problem?

Thanks.

Peter

@m1ctk
Copy link

m1ctk commented Nov 22, 2022

It will be some time before there is an update imo. With so much added content I wonder if any new update will cause GIA to crash !

@pebceb
Copy link

pebceb commented Nov 22, 2022

Given that ITV is not working at all it might be worth the risk, but agree we don’t want to risk BBC shows. Scott’s call I guess.

@dvd403
Copy link

dvd403 commented Nov 22, 2022

I’ve just subscribed to Downie 4 and have downloaded all the itv shows I would have obtained through iplayer.Automator. Itvx also has many shows from 4OD

@m1ctk
Copy link

m1ctk commented Nov 22, 2022

Just trying Downie which I've never heard of, very cool ! itvx shouldn't have any shows from all4 though?

The test download was news at ten which wasn't HD res btw

@dvd403
Copy link

dvd403 commented Nov 22, 2022

M1ctk - it does have 4OD - it’s advertised as doing it. Do a search for ‘Grand Designs’ and you’ll find all the shows from series 1 through to 19. 40D itself is up to series 22/23.

@dvd403
Copy link

dvd403 commented Nov 22, 2022

m1ctk

https://www.whattowatch.com/features/itvx-everything-you-need-to-know-about-the-new-itv-on-demand-service

@pebceb
Copy link

pebceb commented Nov 22, 2022

I had a few issues with Downie. Puts the shows as Home Video (easy fix once in apple tv), but I had a few that it downloaded the same episode twice or put the wrong episode number so they were out of order. I also find sometimes I have to put them in Downie twice before they will download. But I manage to work through it, although it gets very confusing sometimes. But at least it's something. Wish I was in the UK, I would just subscribe to the premium service on ITVX. Damn borders.....

@skovatch
Copy link

A few updates: I have figured out how to scan ITVx.com for shows, and can successfully rebuild the cache. That's helpful but doesn't get us much further until yt-dlp can actually download content again.

I also looked into using STV for ITV content and I'm happy to report that I have the 'Get Current Webpage' workflow working. I can get the show metadata into a format that looks good in Apple TV, and downloading it with yt-dlp works perfectly. I will likely need to restore the preferences for ITV quality, though, as they offer a full set of resolutions for most shows (1080p, 720p, down to 384x216) and I doubt you want to download 2gb for every show.

I haven't figured out how to get a full library of STV shows. I'm guessing I'd have to do it category by category like I'm doing for ITV, which will be slow.

I have also been doing a lot of under-the-hood work. I moved a fair amount of code to Swift, added a new logging engine, and removed a lot of duplicated code on the BBC side related to getting metadata and searching. It all feels stable in my usage, so I will likely release that before rolling out any of the ITV/STV changes. I'm trying to be very careful to not break things like your stored auto-record settings and saved queues.

Regarding ITVx and premium content: GiA was never intended to be a tool for downloading subscription-only or DRM-protected material. Managing usernames and passwords is an additional level of complexity that I don't want to deal with, and opens up a whole slew of legal issues that I'm not prepared to handle. That may be a disappointment to some of you but it's just the reality of being a part-time open source project.

@pebceb
Copy link

pebceb commented Nov 22, 2022

Thanks for all your hard work Scott.

@m1ctk
Copy link

m1ctk commented Nov 22, 2022

You have done a excellent job so far and look forward to the finished product :)

@m1ctk
Copy link

m1ctk commented Nov 22, 2022

M1ctk - it does have 4OD - it’s advertised as doing it. Do a search for ‘Grand Designs’ and you’ll find all the shows from series 1 through to 19. 40D itself is up to series 22/23.

You mean All4 of course. Yes but you can't download as it's premium content I noticed

@hugofalkman
Copy link

@m1ctk The whole All4 website is DRM-encrypted, so Downie can't download from it.

Downie has the support for user names and passwords Scott references, but I have found it's only useful for free websites that require a signon. All fee websites I have tried downloading from, Downie says they are DRM-protected, so it can't download.

@m1ctk
Copy link

m1ctk commented Nov 23, 2022

Downie doesn't download in HD with me, should it ? I thought all shows were now HD but I am still getting 896 x 504

@skovatch
Copy link

Sorry, not much to update here. There's discussion about higher resolution downloads and DRM and needing an account. @Vangelis66 has been very helpful here (as usual!)

If you want to follow along with youtube-dl see ytdl-org/youtube-dl#31363 or yt-dlp at yt-dlp/yt-dlp#5570

@hugofalkman
Copy link

After shipping 3 releases fine tuning the ITVX support, Downie yesterday shipped a release with full support for STV. Release notes here https://software.charliemonroe.net/trial/downie/v4/rnotes.html

@skovatch
Copy link

You can now download most(?) ITV content through STV at all resolutions up to full HD. There is some DRM-protected shows that yt-dlp won't touch, and I added guards so the app won't try to download them.

Does anyone know if STV is a subset or superset of ITV?

@Vangelis66
Copy link

@pebceb
Copy link

pebceb commented Jan 16, 2023

I tried an episode of Emmerdale from STV and it worked until the end and then wouldn't put it in Apple TV and started doing errors.

@ejstubbs
Copy link

I've tried the STV download feature with a couple of programmes. In each case I had the following results:

  1. The download got to 100% on the Download Queue pane, but then failed with "Problem Unknown" in the Failed Download Solutions window;
  2. There was a complete, and playable, .mp4 of the programme at the requested resolution in an appropriately named folder in the "TV Shows" subfolder of the "Movies" folder in my user's home folder;
  3. The .mp4 had no subtitles, and there was no .srt file in the programme's folder;
  4. As well as the .mp4 file, there was a .en.dfxp file, and a .jpg file in the folder.

I'd say that's an excellent start, all things considered, and I very much appreciate the work that's been done so far to try to address the issues caused by ITV's decision to implement DRM on their streaming service (especially so with a new series of Endeavour due this year.)

@pebceb
Copy link

pebceb commented Jan 17, 2023

Yes I saw that and was not trying to be critical - just offering feedback. I truly do also appreciate all the work that is being done to fix this.

Cheers.

Peter

@ejstubbs
Copy link

Aye, just reporting what happened on my machine in case it's useful.

Another good thing: ITV shows are now downloadable in 720p - previously they only ever downloaded in sub-SD, which I assume was a limitation of the ITV Hub.

I don't use iTunes for video so didn't see anything relating to that.

One other thing: the shows weren't recorded in the Download History - presumably because they were deemed to have failed downloading. Then again, ISTR seeing something similar with ITV shows in the last few months before ITV Hub became ITVX. Didn't think anything of it at the time, though.

@gacres2
Copy link
Author

gacres2 commented Jan 17, 2023

Does anyone know if STV is a subset or superset of ITV?

Those wikipedia links make it clear that this is a tricky question – neither is the right answer I believe. STV overlaps with ITV, but doesn't carry all ITV programming and STV carries some programs that ITV does not. Also, ITVX has numerous, formerly Britbox, programs that are definitely not part of STV's content.

@Vangelis66
Copy link

there was no .srt file in the programme's folder;
(snipped)
As well as the .mp4 file, there was a .en.dfxp file,

.dfxp is a subs format... As I don't run MacOS myself, I can only offer advice on yt-dlp itself 😜 ...

For STV fetches, the yt-dlp command should explicitly request the vtt subs format; then, ffmpeg is required to convert it to .srt:

yt-dlp [options] "https://player.stv.tv/episode/*" --write-sub --sub-format vtt --convert-subs=srt --embed-subs

Probably GiA has its own routine of embedding a .jpg thumbnail into the .mp4 file, but in standalone yt-dlp, thumbnail embedding is delegated to AtomicParsley (if on PATH) or the (optional) python module mutagen:

...  --write-thumbnail --embed-thumbnail

@skovatch
Copy link

  1. There was a complete, and playable, .mp4 of the programme at the requested resolution in an appropriately named folder in the "TV Shows" subfolder of the "Movies" folder in my user's home folder;

This was due to a bug in how I was passing arguments to AtomicParsley. It should be fixed in v1.25.0. Anything that got added to TV.app is in your Home Movies section.

  1. The .mp4 had no subtitles, and there was no .srt file in the programme's folder;
  2. As well as the .mp4 file, there was a .en.dfxp file, and a .jpg file in the folder.

I apologize, but I am bad about testing subtitle support as I don't use it myself, so I often forget it's there. But thanks to @Vangelis66 and #443 it's a pretty simple fix.

@pebceb
Copy link

pebceb commented Jan 18, 2023

Good morning. I just updated the version and it still does the same thing. Error message and does not go into Apple Tv (not in Home Videos either). Not sure why?

I was able to drag the completed file to Apple TV and then update the data to get it to shows in TV Shows. It did not show in Home Videos for some reason but was in recently added after I dragged and dropped it into Apple TV.

Peter

Screenshot 2023-01-18 at 06 39 37

@skovatch
Copy link

Good morning. I just updated the version and it still does the same thing. Error message and does not go into Apple Tv (not in Home Videos either). Not sure why?

I was able to drag the completed file to Apple TV and then update the data to get it to shows in TV Shows. It did not show in Home Videos for some reason but was in recently added after I dragged and dropped it into Apple TV.

Did you have subtitles turned on? I had no issues without subtitles, but with subtitles enabled it failed because the wrong subtitle format was requested (see above reports). I will push out an update with that fixed soon.

@pebceb
Copy link

pebceb commented Jan 18, 2023

I think I did but am at work right now.

Thanks.

Peter

@ejstubbs
Copy link

I apologize, but I am bad about testing subtitle support as I don't use it myself, so I often forget it's there. But thanks to @Vangelis66 and #443 it's a pretty simple fix.

Firstly: I see no need to apologise for getting something working that was broken through no fault of yours. Secondly - and perhaps pushing my luck somewhat - is there any chance that that fix could be made on a version of GIA which still supports pre-10.15 versions of macOS? I'm still stuck on Mojave until The Powers That Be can provide me with viable alternatives to some critical 32-bit apps (I know, I know...largely out of my control, unfortunately). But it would be handy to have the subtitles working properly...

@pebceb
Copy link

pebceb commented Jan 18, 2023

I turned off subtitles as suggested and STV shows worked flawlessly.

Thanks Scott.

@ejstubbs
Copy link

Just downloaded and installed 1.25.1 - all working perfectly . Many thanks for the effort getting this sorted.

@skovatch
Copy link

skovatch commented Feb 8, 2023

youtube-dl now has the changes needed to get ITVX downloads working again. I tested it and you can download 896x504 programs again. I also have a more robust solution for caching the last 7 days of non-premium ITVX shows. This will match get_iplayer's default of 7 days of program info. As soon as they release an 'official' update of youtube-dl I will work on including it in a future build.

@skovatch
Copy link

skovatch commented Feb 8, 2023

I will keep support for STV downloads there, as it's the only way to get HD programs. I can write a cacher for the last 7 days but I can't guarantee it will pick up everything.

@alphamikegc
Copy link

Many thanks for all your hard work Scott, it's much appreciated. Glad that you are keeping on support for STV, as there are programmes there not available on ITV.

@pebceb
Copy link

pebceb commented Feb 8, 2023

Thanks Scott. really appreciate your ongoing efforts to fix this.

Just curious, do you think eventually you will be able to get HD shows working on ITVX or will it only be the 894x504?

Cheers.

Peter

@skovatch
Copy link

skovatch commented Feb 8, 2023

Just curious, do you think eventually you will be able to get HD shows working on ITVX or will it only be the 894x504?

That will ultimately be up to youtube-dl or yt-dlp. The current thinking I've read is that the only way to get HD content from ITVX is to have a specific client certificate that identifies you as the ITV player on mobile. That would require some significant hacking to extract it, and even then that's only a theory.

@watashi49uk
Copy link

I'm unsure whether or not this is the same problem as previously reported, but when trying to download any programme from ITVX using the "Use Current Webpage" option I receive the message: "Programme Page Not Found -Please ensure the frontmost browser tab is open to an iPlayer episode page or ITV Hub episode page."
The ITVX browser tab is, of course, frontmost and open to an episode page. I'm using a Mac running Monterey Version 12.6.3.

@skovatch
Copy link

skovatch commented Mar 2, 2023

I'm unsure whether or not this is the same problem as previously reported, but when trying to download any programme from ITVX using the "Use Current Webpage" option I receive the message: "Programme Page Not Found -Please ensure the frontmost browser tab is open to an iPlayer episode page or ITV Hub episode page."
The ITVX browser tab is, of course, frontmost and open to an episode page. I'm using a Mac running Monterey Version 12.6.3.

It's indirectly related -- the latest version of GiA disabled Use Current Webpage on itv.com since you can't download anything from it anyway. It will return in a future update when I restore ITV support.

@pebceb
Copy link

pebceb commented Jul 14, 2023

Any update on a fix for ITVX?

Thanks.

Peter

@skovatch
Copy link

Any update on a fix for ITVX?

Sorry, no. I'm waiting on a fix for yt-dlp/yt-dlp#5570 to be merged in, but this is proving to be difficult at best.

@pebceb
Copy link

pebceb commented Jul 17, 2023

Okay thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests