Beebhack
Register
Line 170: Line 170:
 
In fact the DRM appears to be a simple XOR of a large section of the plain vanilla MP4 with a repeating two-byte pattern. Specifically, the first 0x2800 bytes are unchanged, then the next bytes are XOR'd with values beginning 0x3c, 0x53, 0x3c, etc., then the last 0x400 bytes are unchanged. Just to make things interesting the two bytes immediately before the last 0x400 clean ones are XOR'd with reversed values. So (for example) the file is XOR'd with 0, 0,..(0x2800 times).., 0x3c, 0x53, .... ,0x3c, 0x53, 0x3c, 0x53, 0x53, 0x3c, 0, 0, ..(0x400 times)... To recover the original file, just XOR again with these values.
 
In fact the DRM appears to be a simple XOR of a large section of the plain vanilla MP4 with a repeating two-byte pattern. Specifically, the first 0x2800 bytes are unchanged, then the next bytes are XOR'd with values beginning 0x3c, 0x53, 0x3c, etc., then the last 0x400 bytes are unchanged. Just to make things interesting the two bytes immediately before the last 0x400 clean ones are XOR'd with reversed values. So (for example) the file is XOR'd with 0, 0,..(0x2800 times).., 0x3c, 0x53, .... ,0x3c, 0x53, 0x3c, 0x53, 0x53, 0x3c, 0, 0, ..(0x400 times)... To recover the original file, just XOR again with these values.
   
 
(Here is a [http://linuxcentre.net/iplayer_decode Perl script] for XOR decoding the above files)
====== Script to Perform XOR Decoding ======
 
Here is a [http://linuxcentre.net/iplayer_decode XOR decoder Perl script] for above video encoding
 
   
 
===== Overview of video request process =====
 
===== Overview of video request process =====

Revision as of 12:29, 9 June 2008

Purpose

This site describes how to re-purpose BBC online content. To utilise material already paid for by the licence payers in a manner that suits the end user.

Origin / History

This wiki was started as an attempt to bring together the information that was being discovered about the many great BBC services, particularly the iPlayer and particularly after the iPhone beta service was launched.

There are many very useful discussions occurring about the iPhone version of iPlayer and as a result, a great deal of replication of effort. This is an attempt to reduce this replication and organise that information.

Services

Live radio

Podcasts

The BBC podcasts are generally edited highlights of select radio programmes. These are all documented on http://www.bbc.co.uk/radio/podcasts/directory/

Each podcast is encoded at 44KHz 64kbps MP3 and has an RSS 2.0 feed associated with it.

MP3s are available online on the BBC website for a period of 7 days from date published. Some Podcast series are not governed by this 7 day rule, but there aren't many. Also, some of the Podcasts are restricted to the UK only.

There is a Podcasting feed created that lists all current Podcasts available for download as well.

URL: http://downloads.bbc.co.uk/podcasts/ppg.xml

Schema: http://downloads.bbc.co.uk/podcasts/ppg.xsd

The feed is updated every 30 minutes and is a simplified data dump of the internal Podcast database (well, a very stripped down version). It is used internally to create feeds, one of them being the feed that powers the BBC Podcasts directory. The feed also includes data on those which are UK only.

It should be noted that Podcasts that include the "<network id="test" name="Test Network"/>" tag/info should really be ignored as those are just random data Podcasts. They're used internally for testing, hence the network id being "test".

Other variables to look out for are active and public attribute in the programme tag. Active here just tells you if the Podcast series is actually still running at this current point in time. Because not all programmes run through out the year (e.g. The Chequered Flag which is about F1 racing) then the active flag is set to false when the programme series ends.

The public attribute is really there for excluding things from Podcast directory or Podcasts that are currently being worked on but not ready yet. Again, some of these would be experimental and so might be full of random data that would not make sense. Think of it as "nothing to see here, please move along" flag but feel free to take a look at it anyway as it is on a public facing server :)

News / weather data

iPlayer Radio (was 'Listen Again')

While replaying most programmes, there is a 'listen using stand-alone real player' link. This is a link to a .ram file, eg. http://www.bbc.co.uk/radio/aod/shows/rpms/radio4/anyanswers.ram . This file merely contains the address of the stream, served using rtsp, eg. rtsp://rmv8.bbc.net.uk/radio4/sat1402.ra (this link sometimes contains start and end times for the program).

The website http://dave.org.uk/streams/ allow you to browse all the programs on listen again, and get a direct link to the stream. Similar is http://www.comfyyoghurt.co.uk/bbc_streams/ provides easy browsing for mobile browsers.

Once you have the link to the stream, you can save it, using the mplayer -dumpstream . This will play back the stream in real time, saving the file. Programs such as Net Transport and streambox vcr allow you to save rtsp streams faster, by opening multiple connections simultaneously. Avoid flashget. I know of no such programs for linux, suggest being patient with mplayer.

ListenLater is an extension for Firefox that saves iPlayer Radio audio streams as MP3.

Many non music programs on the radio 4 and world service offer podcasts (see above) - download mp3, so check for this first.

iPlayer TV

General technical overview

Programme IDs

All programme episodes on BBC TV are assigned a programme ID (PID) made up of lower-case letter and numbers of around 8 characters. More information on the format of the PID's can be found at the BBC Programme Guide. For Instance for Torchwood, you can see which programmes are available on iPlayer.

In addition to this, it is possible for episodes in iPlayer to have several published "versions". Every episode of a programme should have at least one version - the "original". Other versions that exist are:

version_type_id name
AudioDescribed Audio description
Lengthened Lengthened
OpenSubtitled Open subtitles
Original Original version
Other Other
Shortened Shortened
Signed Sign language
WarningsHigher Stronger content warnings
WarningsLower Milder content warnings
WarningsNone No content warnings
Retrieving programme meta-data

Detailed metadata can be retrieved from XML resources available for each episode with URLs of the form http://www.bbc.co.uk/iplayer/metafiles/episode/<PID>.xml

This contains information on the episode versions available and descriptive information on episodes including still image URLs. This information appears to also specify some of the "More Like This" programs.

3rd Party iPlayer Scripts / Programs

See also List of known interoperable scripts / programs

Windows P2P version

[platform controversy]

Articles on DRM schemes

Flash / RTMP stream version

The web-based Adobe Flash version of the iPlayer was introduced on the 13th of December 2007, designed to bring the iPlayer to a larger audience than the Windows XP only iPlayer.

http://www.theregister.co.uk/2007/12/13/flash_iplayer_launch/

The service streams the video over Abobe's proprietary RTMP, although the video itself is DRM-free.

Video URL resolving process

The flash client goes through the following process to resolve the location of each episode ID video:

  1. The media selector URL for the relevant episode is requested from http://www.bbc.co.uk/mediaselector/3/stream/check/iplayer?pid=[PID] this gives the script:
    1. token
    2. server
    3. identifier
  2. These make up an authentication string [auth] of the form auth=[token]&aifp=v001&slist=[identifier]
  3. The server string is used to fetch ident data from http://[server]/fcs/ident which contains the IP address of the media server
  4. The final stream URL is then resolved as rtmp://[ip_address]:1935/ondemand?_fcs_vhost=[server]&[auth]

An example of this algorithm can be found in this PHP script: http://www.strawp.net/files/download/iplayer_url.zip

At time of writing only Adobe products such as Flash support RTMP, although it is currently on the libcurl TODO list.

iPhone / H.264 over HTTP version

On the 7th of March 2008 the BBC launched an iPhone compatible version of their web-based iPlayer service. Visitors to iPlayer site using an iPhone would be presented with a slightly tweaked interface, but crucially where users of the site using a desktop web browser, iPhone users would see an embedded Quicktime player streaming H.264 video (referred to on iPlayer as MP4) over a normal HTTP protocol.

User-Agent checking

It was quickly spotted by web developers everywhere that all the site was doing to identify iPhone users was inspect the "User-Agent" string sent by the client. By altering this string using freely available browser plugins, anyone could access this feed and view the "MP4" location, opening it any application they wished.

This "bug" was widely recognised as first published by user "Irregular Shed" on Flickr: http://www.flickr.com/photos/twindx/2316284105/

Following this observation, countless developers created their own scripts to pull down iPlayer content.

[BBC charter controversy]

BBC response to non-iPhone clients using the service

On the 13th of March 2008 the BBC released a "fix" designed to stop non iPhone users downloading the H.264 video files:

Although the BBC did not announce the nature of their "fix", it was quickly uncovered by monitoring network traffic between Apple devices and the iPlayer servers. Whereas before the service had only checked for the iPhone/iPod browser's User-Agent HTTP header, it was now also checking the distinctive User-Agent and Range headers sent by the media subsystem. Once this was added to existing scripts, they continued to function largely as normal. By the end of that day, a few developers had already updated their scripts to work around the update:

Update to video file encoding as of 6th June 2008

From early June 2008, the BBC changed the mechanism used to encode their videos for the iPhone/iPod Touch. Although downloading via spoofing still works, the downloaded files will not play on any common desktop software. It is possible that a DRM scheme has been applied to the previously unencrypted H.264 video.

A look at the video file headers suggests that the audio is now encoded by libfaac (1.26.1 unstable). Although no DRM scheme has been formally identified, the most likely candidate is Fairplay, which is built into QuickTime and the iPhone.

Update to video file encoding as of 8th June 2008

In fact the DRM appears to be a simple XOR of a large section of the plain vanilla MP4 with a repeating two-byte pattern. Specifically, the first 0x2800 bytes are unchanged, then the next bytes are XOR'd with values beginning 0x3c, 0x53, 0x3c, etc., then the last 0x400 bytes are unchanged. Just to make things interesting the two bytes immediately before the last 0x400 clean ones are XOR'd with reversed values. So (for example) the file is XOR'd with 0, 0,..(0x2800 times).., 0x3c, 0x53, .... ,0x3c, 0x53, 0x3c, 0x53, 0x53, 0x3c, 0, 0, ..(0x400 times)... To recover the original file, just XOR again with these values.

(Here is a Perl script for XOR decoding the above files)

Overview of video request process

The process for requesting the H.264 video is as follows:

  1. Video page is requested and a "BBC-UID" cookie is set, containing the URL-encoded User-Agent string of the browser.
  2. The iPhone version of iPlayer renders a Quicktime object into the page which requests the video URL as http://www.bbc.co.uk/mediaselector/3/auth/iplayer_streaming_http_mp4/[PID]?[RAND] where [PID] is the programme version ID and [RAND] is a random number, although the random number is apparently unused by the service. This URL must have:
    1. The Quicktime User-Agent header
    2. The BBC-UID cookie
    3. A "Range" header of "0-1"
  3. The video URL then 302 redirects to a much more complex URL. The same headers are sent
    1. BBC-UID cookie
    2. Quicktime User-Agent
    3. The "Range" header, this time using "0-"
Example HTTP headers sent by an iPhone session

GET www.bbc.co.uk/mediaselector/3/auth/iplayer_streaming_http_mp4/b009gbsn HTTP/1.1
Accept: */*
Cookie: BBC-UID=74a79c638cbe4c2d416e453f70f0dcf063d0b78d20c0c1d4843ffaaf85b17c9c0Mozilla%2f5%2e0%20%28iPhone%3b%20U%3b%20CPU%20like%20Mac%20OS%20X%3b%20en%29%20AppleWebKit%2f420%2e1%20%28KHTML%2c%20like%20Gecko%29%20Version%2f3%2e0%20Mobile%2f4A93%20Safari%2f419%2e3
User-Agent: Apple iPhone v1.1.4 CoreMedia v1.0.0.4A102
Connection: close
Range: bytes=0-1
Host: www.bbc.co.uk

This is then followed by a redirect response and the client creates another request to the actual media location:

GET download.iplayer.bbc.co.uk/iplayer_streaming_http_mp4/120535810311891748.mp4?token=iVXfwJl4SNsma1ByHhlxO%2FVjsCCjA%2FSSsqr6KzU2p1ymDTC%2Bq94kpfjpz%2BlMpki3epI%2BcfHySRmU%0Aos7y%2BxQGsdNDF2UeI124qoN%2F3b7QmLqZtOuii4bpt7jOVQp7b%2BbFKVM%3D%0A HTTP/1.1
Cookie: BBC-UID=74a79c638cbe4c2d416e453f70f0dcf063d0b78d20c0c1d4843ffaaf85b17c9c0;
User-Agent: Apple iPhone v1.1.4 CoreMedia v1.0.0.4A102
Connection: close
Host: download.iplayer.bbc.co.uk
Range: bytes=0-
A Very Quick "one-liner" download script

The following script is technically one line of shell script. It requires curl, cut, grep and sed:

curl -b cookies.txt -A "Apple iPhone v1.1.3 CoreMedia v1.0.0.4A93" -H "Range: bytes=0-" -o iplayer_download.mov \
  $( \
    curl -i -c cookies.txt -A "Apple iPhone v1.1.3 CoreMedia v1.0.0.4A93" -H "Range: bytes=0-1" \
      $( \
        curl -i -A "iPhone, LOL" -c cookies.txt $1 \
        | grep "pid       :" \
        | cut -d\' -f2 \
        | sed "s/\([0-9a-z]\)/http:\/\/www.bbc.co.uk\/mediaselector\/3\/auth\/iplayer_streaming_http_mp4\/\1/" \
      ) \
    | grep Location \
    | cut -d: -f2,3 \
  ) 

If it's saved off to a script file, pass it an web iPlayer episode page and it will save the episode to a file named "iplayer_download.mov".

BBC tightening of User Agent check

On 7th April 2008 it was noticed by "Irregular Shed" that the BBC have tightened up the user agent checking required to convince iPlayer that it is talking to an iPhone/iPod Touch. Whereas previously it was sufficient to pass nothing more than "iPhone" as a user agent string to access the service, it now checks for more. To be on the safe side, all scripts should make use of the full, verbose user agent string:

Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/419.3

... or the iPod Touch equivalent:

Mozila/5.0 (iPod; U; CPU like Mac OS X; en) AppleWebKit/420.1 (KHTML, like Gecko) Version/3.0 Mobile/3A101a Safari/419.3

In addition, there may now be multiple pid values in the HTML, yielding different versions of the video with different availability periods: a script must choose the right one in order to download successfully. Note that it is recommended to use the episode metadata XML as opposed to screen scraping the player page as an update in page code may break a script based on screen scraping, whereas official BBC clients rely on the metadata XML. See section on Programme IDs.

List of known interoperable scripts / programs

iPlayer for Nintendo Wii

On 9th April 2008, the BBC announced that a beta version of iPlayer for the Nintendo Wii was in late stages on development, with a beta version available immediately and a more polished version later in the year. It will appear within the Internet channel of the Wii's UI.

The Wii version is more of a tweak to the existing web version than the iPhone specific version is as the Wii Internet Channel is simply a version of the Opera Browser (version 9.00 upwards) The changes are as follows:

  • The server detects the browser User-Agent and adds a javascript flag "_WII = true" to the inline javascript on the programme view page
  • The javascript file iplayer_info.js uses this flag in a few places to determine the lifetime of the streamed media, but mostly just to fall back to a flash 7 compatible stream
  • The iPlayer client is loaded as normal with these variables.

The stream appears to use the RTMP protocol as with the normal browser client, as opposed to an HTTP connection which the iPhone uses and is in the lower quality Sorenson Spark codec as opposed to the ON2 VP6 codec used in later Flash versions. More technical information is on Anthony Rose's blog.

The User-Agent used by the Production Nintendo Wii is:

Opera/9.10 (Nintendo Wii; U; ; 1621; en)
Alternative Wii Interface

An unofficial alternative interface for browsing iPlayer on the Wii, named WiiPlayer, was released on May 28th. The interface is designed to overcome the usability problems that arise when using the standard iPlayer webpage on the small fixed width resolution of Wii Opera. WiiPlayer is optimised for standard definition TVs and styled for better integration with the existing Wii UI.

To use this interface point your Wii browser at: http://defaced.co.uk/wiiplayer/