I think I know why the original test might have worked.
Based on the 15 min recording which contained 10119183 blocks of data. 15 min of recording will have about 15 X 60 x 25 frames = 22500 frames. Adding 10000 data blocks only added about 22 frames of data or less than 1 sec of video.
Going to try treating the non standard sync byte as a simple encryption key by using it to bitwise manipulate the packet data using in turn AND OR and XOR. Any other suggestions gratefully received.
Can you tell if there is any relation between the header and what happens to the sync byte? I.e, does the header change radically at the start of an encrypted packet? (And for info, how long is the sequence of nonstandard sync bytes?) If there is, the key (pun not intended ) must lie there rather than with the sync byte I'd think.
Can you tell if there is any relation between the header and what happens to the sync byte? I.e, does the header change radically at the start of an encrypted packet? (And for info, how long is the sequence of nonstandard sync bytes?) If there is, the key (pun not intended ) must lie there rather than with the sync byte I'd think.
I will have a look to see if there is a recognisable pattern
I will have a look to see if there is a recognisable pattern
Finally got round to this but not sure what if anything it shows. Heres the results of the first 30 packets of an encrypted file, first 4 columns the junk that's normally stripped out and the 5th the non standard sync byte
Finally got round to this but not sure what if anything it shows. Heres the results of the first 30 packets of an encrypted file, first 4 columns the junk that's normally stripped out and the 5th the non standard sync byte
I finally opened the box, connected the disk to a sata to usb external drive box and pulled some HD stuff - the new Wallace & Gromit cartoon recorded at Christmas, the BBC HD test card, and a 30 sec recording of the preview when I had simultaneously recorded on Mythtv. Both the 30 sec clip and the W&G cartoon are fully encrypted. The test card has a clear section of 141846 packets from when I started recording (during the announcement caption), and then goes encrypted.
The Broadcom system chip that some other post says is the guts of the HDR contains a hardware crypto engine that does several algorithms - DES, AES & DVB Common Scrambling Algorithm amongst them. So I surmise that the 192-byte packets are passed through this before recording. It would take some much deeper analysis of the HDR code, I suspect, to get much further and I won't be engaging in that.
. It would take some much deeper analysis of the HDR code, I suspect, to get much further and I won't be engaging in that.
Thank's for the expert insight. I am happy now I can simply save HD (archive for future watching). There's so many fta HD boxes that have no truck with HD encryption that to anyone who this is a problem they can simply buy one of those.
Funnily enough I was doing some digging today and found out a bit about mpeg scrambling systems. I came to the conclusion that this was well beyond me, and even if it wasn't it would probably make somebody somewhere very unhappy!
As mpeg is a lossy format, I wonder what sort of degradation of the picture scrambling involves? Can't say I noticed anything myself.
Thank's for the expert insight. I am happy now I can simply save HD (archive for future watching). There's so many fta HD boxes that have no truck with HD encryption that to anyone who this is a problem they can simply buy one of those.
As mpeg is a lossy format, I wonder what sort of degradation of the picture scrambling involves? Can't say I noticed anything myself
Mpeg might be lossy but only at the encoding stage, doubt if the encryption is. Decrypting the encrypted data should give a byte for byte accurate copy of the original unencrypted data stream.
I've done some of my own tests with the encrypted hd ts files with no luck in playing them back. I created a dos program that read in a ts file (any size), and repaced the sync bytes with a valid sync, it could also strip the 4 bytes of junk at the beginning of the 192 byte packet to recreate a new ts file with only 188 byte packets :-) meant I could use then analyze the ts stream with a program that only supported 188 byte ts packets :-), program works a treat! did however prove to me that the encrypted hd ts files contain more than just junk sync bytes, PIDs and sequence counters! the lost is definetly encrypted
Well done on your efforts, but as you can see from the thread you have been reinventing the wheel. But it's still a square wheel since there is hardware based encryption, whose effects you have discovered, and which would take a good deal more knowledge and determination to solve than it seems anyone here is able to bring.
Well done on your efforts, but as you can see from the thread you have been reinventing the wheel. But it's still a square wheel since there is hardware based encryption, whose effects you have discovered, and which would take a good deal more knowledge and determination to solve than it seems anyone here is able to bring.
Welcome to the future.
It's more a case of being willing rather than being able, though extracting code & reverse engineering it is definitely not for beginners! It does seem pretty silly though that the Freesat agreement (presumably) forces this on the manufacturers, since the stuff is FTA anyway and is easily recorded with non-Freesat kit.
It does seem pretty silly though that the Freesat agreement (presumably) forces this on the manufacturers, since the stuff is FTA anyway and is easily recorded with non-Freesat kit.
Yes, especially if/when the non- Freesat twin tuner boxes with 7 day epg materialise.
As it is, I have Freeview for SD, so I can't see only having one tuner with the 7 day epg becoming a problem for quite a long time.
Son_t, many thanks for this link, it's solved the problem!
The link mentions many programs that I had tried previously (TsRemux, MPEG Streamer, Super, Total Video Converter) all of which don't work on the Foxsat ts files, but then there was one mentioned that I hadn't tried: MediaCoder. This works an absolute treat - and it's free. It can turn a normal length movie ts file into an mpeg in a few minutes, just like VideoReDo. It doesn't re-encode but simply swaps the ts container for an mpeg one, so is very quick.
For those interested in using MediaCoder, just stick to the default and auto settings for all parameters except
* choose 'raw video' as the video format and 'mpeg2' as the container
* ensure that both audio & video options for both 'enable' and 'copy' are ticked
* in the 'picture' tab, choose 16:9 ratio.
A one hour SD broadcast takes under 5 minutes to convert.
Have you upgraded to MediaCoder 0.7.2.4535 yet, Nick? I'm having similar struggles with what should be a simple task (why can't Humax give us files in a format that transfers easily to other devices?) and found even the MediaCoder "Setup Wizard" horrendously overcomplicated for a non-expert.
In this version, for a lot of the settings no default value is shown, and some of the options you mentioned (such as "raw video") aren't there (I chose "raw YUV"). I saw no picture tab, just a series of pages displayed in the web browser. When I tried to follow your instruction above and then attempted to transcode my file, after a few seconds the program popped up the error "Incorrect parameters for FFmpeg. No file is outputted. Last Error Code: 014".
The description for the error is
014
* Description: A video encoder error is encountered
* Possible causes:
o Video encoder cannot be started as expected
o Video encoder is absent
...which tells me nothing useful at all. Any ideas?
Have you upgraded to MediaCoder 0.7.2.4535 yet, Nick? I'm having similar struggles with what should be a simple task (why can't Humax give us files in a format that transfers easily to other devices?) and found even the MediaCoder "Setup Wizard" horrendously overcomplicated for a non-expert.
In this version, for a lot of the settings no default value is shown, and some of the options you mentioned (such as "raw video") aren't there (I chose "raw YUV"). I saw no picture tab, just a series of pages displayed in the web browser. When I tried to follow your instruction above and then attempted to transcode my file, after a few seconds the program popped up the error "Incorrect parameters for FFmpeg. No file is outputted. Last Error Code: 014".
The description for the error is
014
* Description: A video encoder error is encountered
* Possible causes:
o Video encoder cannot be started as expected
o Video encoder is absent
...which tells me nothing useful at all. Any ideas?
No, I didn't upgrade, but I agree that there are too many options for us non-tecchies. I haven't used MediaCoder recently as I decided it was easier to find a piece of kit that could play the ts files - I switched my old media player for the WD TV one from Western Digital, which plays just about every format.
Sorry, I should have mentioned, VLC was one of the first programs I tried. I followed some directions on how to convert ts to mpg using VLC that I found on the web, but no mpg file get created using the Media >convert/save function then adding the mpg extension to the output file. Has anyone been successful using VLC, please?
I find that VLC is the best programme to convert .ts files to MPEG 2. It does not suffer from the audio / video sync problems of some other programmes, and it appears to include error correction, and its free.
However, the function to convert only seems to work in older versions of VLC (I use 0.9.7 -- the current version 1.0.3 gives very blocky output).
To download 0.9.7, you need to google "old vlc"
To convert the file, choose midia> convert/save. Select the input file. Check the "file" box and enter the filename. On the Encapsulation tab, select "MPEG-PS", then click "SAVE".
I find that VLC is the best programme to convert .ts files to MPEG 2. It does not suffer from the audio / video sync problems of some other programmes, and it appears to include error correction, and its free.
However, the function to convert only seems to work in older versions of VLC (I use 0.9.7 -- the current version 1.0.3 gives very blocky output).
To download 0.9.7, you need to google "old vlc"
To convert the file, choose midia> convert/save. Select the input file. Check the "file" box and enter the filename. On the Encapsulation tab, select "MPEG-PS", then click "SAVE".
Job done
Thanks for the tip - I'm using the latest VLC version OK now, so I won't be going back to v 0.9.7. I discovered that for some reason I had to set the destination file to the root directory c:\ to get it to work, it didn't like writing into subfolders for some reason. Very odd, I don't see anyone else reporting this problem, but it's not a hassle and I just move the mpg file once it's created.
I'm not expert in this field but I have met with some success in converting .ts files to .mpg files and burning them to dvd as a Linux user. The version of Linux I use is Ubuntu 9.10, codenamed Karmic Koala. I have all standard repositories and the very useful Medibuntu repository enabled.
I made an external usb hd by fitting an 80GB 2.5" drive taken from a laptop I was upgrading into a 'no name' external usb hd enclosure. To avoid the file size limitations of common Microsoft operating systems I used the GParted utility to format the newly put together hd as ext3.
I plugged this ext3 formatted usb drive into the Humax Foxsat PVR (With latest off air upgrade installed.) After a few moments the usb drive was recognised in the 'media explorer' and it was straightforward to copy a largish video program to the hd, which I then disconnected. I got an on screen message saying the usb device had been removed.
I was astonished to find the 'file' had arrived as a ts file plus two support files of the same name. Googling confirmed that the ts file (the largest one) was the one of interest. So how to convert the ts file to something more familiar? This is what I did:-
1) I plugged the external usb hd into my trusty Ubuntu PC
2) The WinFF utility has a useful series of Presets. I chose DVD and then PAL DVD HQ Widescreen. The conversion is reasonably quick.
3)The Avidemux video editor will edit (ie top & tail & generally edit) mpg files WITHOUT TRANSCODING, so it is lightning fast (3 minutes to process and re-save a topped-and-tailed 2.7 GB mpg video file. ) I chose Video Copy, Audio copy and Format MPEG-PS(A+V).
4)Once I had the mpeg as I wanted it I used the dvdstyler easy to use dvd authoring software ro burn the file to a dvd iso file. You can easily create a top menu using this friendly software.
5)Having satisfied myself that the iso was ok (play in vlc media player) I used the legendary k3b burner software to burn the dvd. The latest version of the Brasero dvd burner is a little flakey on mu hardware but k3b works like a champ.
6)Tested the dvd in a dvd player -success!
I have three further points to make.
1)All the software I used comes free, as in no charge.
2)This fairly sparse walkthrough is for those familiar with Ubuntu Linux (Or Debian, possibly). It *is* possible to install Ubuntu on a Windows machine in a dual boot setup. If you want to learn a new operating system, then fine -go ahead but remember that I'm too long in the tooth to offer any hand-holding facility: bluntly, I have no intention of spending the rest of my life helping folk learn Ubuntu. There are plenty of forums better equipped than I am to do that!
3)This *is* just a sparse walkthrough. I'm no expert and entirely self taught. What I have described worked for me. I hope it works for you.
Comments
Based on the 15 min recording which contained 10119183 blocks of data. 15 min of recording will have about 15 X 60 x 25 frames = 22500 frames. Adding 10000 data blocks only added about 22 frames of data or less than 1 sec of video.
I will have a look to see if there is a recognisable pattern
Finally got round to this but not sure what if anything it shows. Heres the results of the first 30 packets of an encrypted file, first 4 columns the junk that's normally stripped out and the 5th the non standard sync byte
I finally opened the box, connected the disk to a sata to usb external drive box and pulled some HD stuff - the new Wallace & Gromit cartoon recorded at Christmas, the BBC HD test card, and a 30 sec recording of the preview when I had simultaneously recorded on Mythtv. Both the 30 sec clip and the W&G cartoon are fully encrypted. The test card has a clear section of 141846 packets from when I started recording (during the announcement caption), and then goes encrypted.
The Broadcom system chip that some other post says is the guts of the HDR contains a hardware crypto engine that does several algorithms - DES, AES & DVB Common Scrambling Algorithm amongst them. So I surmise that the 192-byte packets are passed through this before recording. It would take some much deeper analysis of the HDR code, I suspect, to get much further and I won't be engaging in that.
Thank's for the expert insight. I am happy now I can simply save HD (archive for future watching). There's so many fta HD boxes that have no truck with HD encryption that to anyone who this is a problem they can simply buy one of those.
As mpeg is a lossy format, I wonder what sort of degradation of the picture scrambling involves? Can't say I noticed anything myself.
Absolutely! That's what I did.
Mpeg might be lossy but only at the encoding stage, doubt if the encryption is. Decrypting the encrypted data should give a byte for byte accurate copy of the original unencrypted data stream.
Now, what were you saying about which encryption algorithms are used and where they keys are?
I've done some of my own tests with the encrypted hd ts files with no luck in playing them back. I created a dos program that read in a ts file (any size), and repaced the sync bytes with a valid sync, it could also strip the 4 bytes of junk at the beginning of the 192 byte packet to recreate a new ts file with only 188 byte packets :-) meant I could use then analyze the ts stream with a program that only supported 188 byte ts packets :-), program works a treat! did however prove to me that the encrypted hd ts files contain more than just junk sync bytes, PIDs and sequence counters! the lost is definetly encrypted
Of course not...
Well done on your efforts, but as you can see from the thread you have been reinventing the wheel. But it's still a square wheel since there is hardware based encryption, whose effects you have discovered, and which would take a good deal more knowledge and determination to solve than it seems anyone here is able to bring.
Welcome to the future.
It's more a case of being willing rather than being able, though extracting code & reverse engineering it is definitely not for beginners! It does seem pretty silly though that the Freesat agreement (presumably) forces this on the manufacturers, since the stuff is FTA anyway and is easily recorded with non-Freesat kit.
Yes, especially if/when the non- Freesat twin tuner boxes with 7 day epg materialise.
As it is, I have Freeview for SD, so I can't see only having one tuner with the 7 day epg becoming a problem for quite a long time.
Have you upgraded to MediaCoder 0.7.2.4535 yet, Nick? I'm having similar struggles with what should be a simple task (why can't Humax give us files in a format that transfers easily to other devices?) and found even the MediaCoder "Setup Wizard" horrendously overcomplicated for a non-expert.
In this version, for a lot of the settings no default value is shown, and some of the options you mentioned (such as "raw video") aren't there (I chose "raw YUV"). I saw no picture tab, just a series of pages displayed in the web browser. When I tried to follow your instruction above and then attempted to transcode my file, after a few seconds the program popped up the error "Incorrect parameters for FFmpeg. No file is outputted. Last Error Code: 014".
The description for the error is
014
* Description: A video encoder error is encountered
* Possible causes:
o Video encoder cannot be started as expected
o Video encoder is absent
...which tells me nothing useful at all. Any ideas?
No, I didn't upgrade, but I agree that there are too many options for us non-tecchies. I haven't used MediaCoder recently as I decided it was easier to find a piece of kit that could play the ts files - I switched my old media player for the WD TV one from Western Digital, which plays just about every format.
I find that VLC is the best programme to convert .ts files to MPEG 2. It does not suffer from the audio / video sync problems of some other programmes, and it appears to include error correction, and its free.
However, the function to convert only seems to work in older versions of VLC (I use 0.9.7 -- the current version 1.0.3 gives very blocky output).
To download 0.9.7, you need to google "old vlc"
To convert the file, choose midia> convert/save. Select the input file. Check the "file" box and enter the filename. On the Encapsulation tab, select "MPEG-PS", then click "SAVE".
Job done
I made an external usb hd by fitting an 80GB 2.5" drive taken from a laptop I was upgrading into a 'no name' external usb hd enclosure. To avoid the file size limitations of common Microsoft operating systems I used the GParted utility to format the newly put together hd as ext3.
I plugged this ext3 formatted usb drive into the Humax Foxsat PVR (With latest off air upgrade installed.) After a few moments the usb drive was recognised in the 'media explorer' and it was straightforward to copy a largish video program to the hd, which I then disconnected. I got an on screen message saying the usb device had been removed.
I was astonished to find the 'file' had arrived as a ts file plus two support files of the same name. Googling confirmed that the ts file (the largest one) was the one of interest. So how to convert the ts file to something more familiar? This is what I did:-
1) I plugged the external usb hd into my trusty Ubuntu PC
2) The WinFF utility has a useful series of Presets. I chose DVD and then PAL DVD HQ Widescreen. The conversion is reasonably quick.
3)The Avidemux video editor will edit (ie top & tail & generally edit) mpg files WITHOUT TRANSCODING, so it is lightning fast (3 minutes to process and re-save a topped-and-tailed 2.7 GB mpg video file. ) I chose Video Copy, Audio copy and Format MPEG-PS(A+V).
4)Once I had the mpeg as I wanted it I used the dvdstyler easy to use dvd authoring software ro burn the file to a dvd iso file. You can easily create a top menu using this friendly software.
5)Having satisfied myself that the iso was ok (play in vlc media player) I used the legendary k3b burner software to burn the dvd. The latest version of the Brasero dvd burner is a little flakey on mu hardware but k3b works like a champ.
6)Tested the dvd in a dvd player -success!
I have three further points to make.
1)All the software I used comes free, as in no charge.
2)This fairly sparse walkthrough is for those familiar with Ubuntu Linux (Or Debian, possibly). It *is* possible to install Ubuntu on a Windows machine in a dual boot setup. If you want to learn a new operating system, then fine -go ahead but remember that I'm too long in the tooth to offer any hand-holding facility: bluntly, I have no intention of spending the rest of my life helping folk learn Ubuntu. There are plenty of forums better equipped than I am to do that!
3)This *is* just a sparse walkthrough. I'm no expert and entirely self taught. What I have described worked for me. I hope it works for you.
Uwe907