PBO File Format: Difference between revisions
Lou Montana (talk | contribs) m (Text replacement - "<br/>" to "<br>") |
m (category) |
||
(20 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
{{ | {{Feature|UnsupportedDoc}} | ||
{{TOC|side}} | |||
'''PBO''' means "'''p'''acked '''b'''ank '''o'''f files". A '''.pbo''' is identical in purpose to a zip or rar. It is a container for folder(s) and file(s). | |||
The engine will internally expand any '''*.pbo''' back out to its original, tree-folder, form. | |||
==== Pbo .ext types ==== | |||
generically called 'pbo' there are in fact different extension names employed, dependent on the engine being used. In all cases, the contents ultimately are identical to pbos. | |||
A | *ifa: A World War2 mod hard wired to only accept ifa files. | ||
*ebo: Used by vbs initially, and bis later, to encrypt the contents of a pbo and call it a different extension. | |||
*xbo: Used by vbsLite encrypt with a different algorithm to above. | |||
** the encryption method used for vbslite UK and vbslite US is different. | |||
=== Legend === | |||
See [[Generic FileFormat Data Types]]. | |||
=== Compression === | |||
In addition to simply packaging all files and folders in a tree into a single file, some, all, or none of the files within can be compressed using Apple's method of LZSS compression. Which type of files are compressed is ''entirely'' optional. Ones that won't benefit normally are not and some files in the Arma engines cannot be because bis forgot they had this code. The intent behind compression '''was''' for internet use and, in the 'good old days', simply to reduce hard disk storage requirements. The ''actual'' use of compression is becoming less 'popular' as it does represent a small load on the engine. Operation Flashpoint: Elite cannot work with ''compressed .pbo'' files - see [[#{{ofpe}} PBO|Elite PBO]]s. | |||
In addition to simply packaging all files and folders in a tree into a single file, some, all, or none of the files within can be compressed. Which type of files are compressed is ''entirely'' optional. | |||
=== | === PboTypes === | ||
==== Mission Versus Addon ==== | |||
Across all engines there are two distinct types of pbo. | |||
* Mission pbos do not contain config.cpps and live in a mission folder (be it campaign, single, or multiplayer). They have no use for a pboPrefix, nor a Properties header. | |||
* Addon pbos contain a config.cpp/bin which at-the-least identifies the addon name, and, with the exception of the initial cwc engine, rely on a pbo properties header. They live in an addons\ folder. | |||
==== pbo engine differences ==== | |||
*Cold War Crisis: the very first in the series (disregarding the 'proof of concept' release a year earlier) has no pboProperties header and no checksums. It does however form the core of all other engines. It's format has never been altered | |||
*Resistance: Added a Properties header as the first 'entry' of the pbo. | |||
*Vbs1: same as Resistance (but encrypted). | |||
*Xbox Elite: Added a 0 plus 4 byte checksum to end of file. | |||
*Arma: Changed elite's checksum to a 0 plus 20 byte sha key to end of file. | |||
*VBS2: same as arma (but encrypted) | |||
*Dayz: same as arma | |||
Note also that in almost all cases both the properties header and/or the sha key are optional. | |||
== Main format == | |||
The format of a .pbo contains: | |||
#a header consisting of 21byte contiguous file name structures called 'entries'. The very first entry might exceed 21 bytes (see below) | |||
#one, contiguous data block. | |||
#an (optional) 5 byte checksum (Elite) or a 21 byte signature key (Arma) | |||
With exceptions, each entry defines a file contained in the .pbo, its size, date, name, it is whether it is compressed. | |||
Because entries and data are contiguous, there is no need for an offset to the 'next' file. <u>Every</u> file, even zero-length ones, are recorded in the header. | |||
However, note that there is no provision for, and no ability to, store empty folders. Folders as such are indicated simply by being part of the filename. There are no, folder entries, and consequently, empty folders, cannot be included in a .pbo because there is no filename associated with them. Put another way, an empty folder, if it could be stored (and it can't), would appear to be an empty filename when dePbo'd. | |||
The last header 'entry' is filled with zeroes. The next byte is the beginning of the data block. | |||
== PBO Header Entry == | |||
A standard .pbo entry as follows: | |||
<syntaxhighlight lang="cpp"> | |||
struct entry | |||
{ | |||
Asciiz filename; // a zero terminated string defining the path and filename, | |||
// ''relative to the name of this .pbo'' or it is prefix. | |||
// Zero length filenames ('\0') indicate the first (optional), or last (non optional) entry in header. | |||
// Other fields in the last entry are filled by zero bytes. | |||
char[4] MimeType; // 0x56657273 'Vers' properties entry (only first entry if at all) | |||
// 0x43707273 'Cprs' compressed entry | |||
// 0x456e6372 'Enco' comressed (vbs) | |||
// 0x00000000 dummy last header entry | |||
ulong OriginalSize; // Uncompressed: 0 or same value as the DataSize | |||
// compressed: Size of file after unpacking. | |||
// This value is needed for byte boundary unpacking | |||
// since unpacking itself can lead to bleeding of up | |||
// to 7 extra bytes. | |||
= | ulong Offset; // not actually used, always zeros (but vbs = encryption data) | ||
ulong TimeStamp; // meant to be the unix filetime of Jan 1 1970 +, but often 0 | |||
ulong DataSize; // The size in the data block. | |||
// This is also the file size when '''not''' packed | |||
}; | |||
</syntaxhighlight> | |||
=== Null Entries === | |||
Entries with no file name indicate: | |||
# End of header, content all zeros and ignored regardless. | |||
# PboProperties entry as the very first entry of the file (not present for cwc or any mission.pbo) | |||
=== PboProperties === | |||
<syntaxhighlight lang="cpp"> | |||
struct standard entry | |||
{ | |||
Asciiz filename; // = 0 | |||
char[4] MimeType; // 0x56657273 'Vers' properties entry (only first entry if at all) | |||
ulong OriginalSize; // = 0 | |||
ulong Reserved; // = 0 | |||
ulong TimeStamp; // = 0 | |||
ulong DataSize; // = 0 | |||
} // end of 'standard' entry | |||
struct properties | |||
{ | |||
Asciiz this1, that1; // eg this=that; | |||
}[...]; | |||
byte end; // = 0 | |||
</syntaxhighlight> | |||
There can be as many contiguos paired Strings as, well, as many as, a piece of string! | |||
The '''LAST''' (or only!) String is a zero length {{Wikipedia|Null-terminated string|ASCIIZ}} string (eg. '\0'). | |||
==== {{ofpr}} PBO ==== | |||
<syntaxhighlight lang="cpp"> | |||
struct properties | |||
{ | |||
"product" = "OFP: Resistance" | |||
}; | |||
</syntaxhighlight> | |||
==== {{ofpe}} PBO ==== | |||
<syntaxhighlight lang="cpp"> | |||
struct properties | |||
{ | |||
"prefix" = "Addon\FOLDER\Name" | |||
}; | |||
</syntaxhighlight> | |||
==== Arma PBO ==== | |||
<syntaxhighlight lang="cpp"> | |||
prefix=Addon\FOLDER\Name | |||
version="123" | |||
engine="arma3" | |||
author=I am famous" | |||
anything= else that takes your fancy | |||
</syntaxhighlight> | |||
== | == End of File Checksum or Sha == | ||
=== {{ofp}} === | |||
Not present for {{ofp}} pbos | |||
=== {{ofpe}} === | |||
<syntaxhighlight lang="cpp"> | |||
{ | |||
byte always0; | |||
int checksum; | |||
} | |||
</syntaxhighlight> | |||
=== Arma === | |||
<syntaxhighlight lang="cpp"> | |||
{ | |||
byte always0; | |||
byte sha[20]; //std md5 checksum only used for MP play | |||
} | |||
</syntaxhighlight> | |||
== Data compression == | |||
Data compression is a mild, but effective use of Apple Corps LZSS (a variant of liv-zempel & Huffman), allowing (up to) 8k of previous data to repeat itself. | |||
Compression is indicated when a signature of 0x43707273 '''and''' the file sizes do not match in the entry. | |||
The following code <u>also</u> applies to the packing method employed in wrp (OPRW) and pac/paa files which have no header info simply a block of known output length that must be decoded. | |||
In all cases, the OUTPUT size is known. With .pbo's, the INPUT size is only a boundary definition to the next block of compressed data. | |||
It is not used or relevant to decoding data because (up to) 7 residual bytes could exist in the last flag word of the block. | |||
As such, only the fixed in concrete output size is relevant. | |||
The compressed data block is in contiguous 'packets' of different lengths: | |||
<syntaxhighlight lang="cpp"> | |||
block {packet1}...{packetN} {4 byte checksum} | |||
packet | |||
{ | |||
byte flagbits; | |||
byte packetdata[...]; // no fixed length | |||
} | |||
</syntaxhighlight> | |||
The contents of the packetdata contain mixtures of raw data that is passed directly to the output, and, 2byte pointers. | The contents of the packetdata contain mixtures of raw data that is passed directly to the output, and, 2byte pointers. | ||
Format: | Format: bit values determine what the packetdata is. It is interpeted '''lsb''' first thus: | ||
BitN = 1 - append byte directly to file (read single byte) | |||
BitN =1 | BitN = 0 - pointer (read two bytes) | ||
BitN= 0 | |||
for example: | for example: | ||
Line 189: | Line 200: | ||
There are three bytes in the block a little further past the format flag that will be passed directly to the output when encountered, and there are FIVE pointers. | There are three bytes in the block a little further past the format flag that will be passed directly to the output when encountered, and there are FIVE pointers. | ||
In this example, first byte of packetdata | In this example, first byte of packetdata is passed to output, 2 bytes are read to make a pointer, next byte is passed (ultimately) to output and so on. | ||
For the very last packet in the block, it is almost inevitable that there will be | For the very last packet in the block, it is almost inevitable that there will be | ||
excessive bits. These are ignored (truncated) as the final output length is always | excessive bits. These are ignored (truncated) as the final output length is always | ||
known from the Entry. You cannot rely on the ignored bits in the format flag (up to seven | known from the Entry. You cannot rely on the ignored bits in the format flag (up to seven | ||
of them) to be any particular value (0 or 1). | of them) to be any particular value (0 or 1) and must be ignored. | ||
A pointer consists of a 12 bits address and 4 bit run length. | A pointer consists of a 12 bits address and 4 bit run length. | ||
The pointer is a reference to somewhere in the previous 4k max of built output. Given Intel's | The pointer is a reference to somewhere in the previous 4k max of built output. Given Intel's endian word format the bytes b1 and b2 form a short word value B2B1 | ||
The format of B2B1 is | The format of B2B1 is unfortunately in Big Endian (motorola) AAAA LLLL AAAAAAAA, requiring some shift and mask fiddling. | ||
The address refers to the start of some data in the currently rebuilt part of the file. It is a value, relative to the current length of the reconstructed part of the file (FL). | The address refers to the start of some data in the currently rebuilt part of the file. It is a value, relative to the current length of the reconstructed part of the file (FL). | ||
The run length of the data to be copied, the 'pattern' has 4 bits and therefore, in theory, 0 to 15 bytes can be duplicated. In practice the values are 3..18 bytes because copying 0,1 or 2 bytes makes no sense. | The run length of the data to be copied, the 'pattern' has 4 bits and therefore, in theory, 0 to 15 bytes can be duplicated. | ||
In practice the values are 3..18 bytes because copying 0,1 or 2 bytes makes no sense. | |||
Relative position (rpos) into the currently built output is calculated as | Relative position (rpos) into the currently built output is calculated as | ||
rpos = FL - ((B2B1 &0x00FF) + (B2B1 & 0xF000)>>4) ) | |||
The length of the data block: rlen | The length of the data block: rlen | ||
rlen = (B2B1 & 0x0F00>>8) + 3 | |||
With the values of rpos and rlen there are three basic situations possible: | With the values of rpos and rlen there are three basic situations possible: | ||
Line 217: | Line 229: | ||
block is added to the end of the file, giving a new length of FL = FL + rlen. | block is added to the end of the file, giving a new length of FL = FL + rlen. | ||
rpos + rlen > FL // data to copy exceeds what's available | rpos + rlen > FL // data to copy exceeds what's available | ||
In this situation the data block has a length of FL – rpos and it is added to the reconstructed file until FL = rpos + rlen. | In this situation the data block has a length of FL – rpos and it is added to the reconstructed file until FL = rpos + rlen. | ||
rpos + rlen < 0 | rpos + rlen < 0 | ||
This is a special case where spaces are added to the decoded file until FL = FL,Initial + rlen | This is a special case where spaces are added to the decoded file until FL = FL,Initial + rlen | ||
The checksum, the last four bytes of any compressed data block. It is an unsigned long (Intel Little Endian order). It is simply a byte-at-a-time, unsigned additive spillover of the '''decompressed''' data. | The checksum, the last four bytes of any compressed data block. It is an unsigned long (Intel Little Endian order). | ||
It is simply a byte-at-a-time, unsigned additive spillover of the '''decompressed''' data. | |||
Each and every compressed data block, contains | Each and every compressed data block, contains its own, unique checksum. | ||
There is no | There is no checksum or other protective device, employed on a .pbo overall. | ||
Exceptions: Elite and Arma have residual data after the end of contiguous data block that do represent a signature for the file. | |||
==Bibliography== | == Bibliography == | ||
* DosTools: http://dev-heaven.net/projects/list_files/mikero-pbodll | |||
* cpbo: http://www.kegetys.net/arma/ | |||
== Open Source PBO Libraries == | |||
===C=== | === C === | ||
* JAPM: https://github.com/RaJiska/JAPM | |||
* armake: https://github.com/KoffeinFlummi/armake | |||
* libpbo: https://github.com/Learath2/libpbo | |||
===C++=== | === C++ === | ||
libpbo: https://github.com/StidOfficial/libpbo | |||
* libpbo: https://github.com/StidOfficial/libpbo | |||
=== C# === | |||
* SwiftPbo: https://github.com/headswe/SwiftPbo | |||
* PboSharp: https://github.com/Shix/PBOSharp | |||
=== Python === | |||
yapbol: https://github.com/overfl0/yapbol <br> | yapbol: https://github.com/overfl0/yapbol <br> | ||
pbo-fuse: https://github.com/Dahlgren/python-pbo-fuse/blob/master/pbo.py | pbo-fuse: https://github.com/Dahlgren/python-pbo-fuse/blob/master/pbo.py | ||
===Java=== | === Java === | ||
ArmaFiles: https://github.com/Krzmbrzl/ArmaFiles | |||
* ArmaFiles: https://github.com/Krzmbrzl/ArmaFiles | |||
===JavaScript=== | === JavaScript === | ||
* Pbo.js: https://github.com/eelislynne/pbo.js | |||
[[Category: | [[Category:BIS File Formats]] | ||
Revision as of 02:16, 25 February 2021
PBO means "packed bank of files". A .pbo is identical in purpose to a zip or rar. It is a container for folder(s) and file(s).
The engine will internally expand any *.pbo back out to its original, tree-folder, form.
Pbo .ext types
generically called 'pbo' there are in fact different extension names employed, dependent on the engine being used. In all cases, the contents ultimately are identical to pbos.
- ifa: A World War2 mod hard wired to only accept ifa files.
- ebo: Used by vbs initially, and bis later, to encrypt the contents of a pbo and call it a different extension.
- xbo: Used by vbsLite encrypt with a different algorithm to above.
- the encryption method used for vbslite UK and vbslite US is different.
Legend
See Generic FileFormat Data Types.
Compression
In addition to simply packaging all files and folders in a tree into a single file, some, all, or none of the files within can be compressed using Apple's method of LZSS compression. Which type of files are compressed is entirely optional. Ones that won't benefit normally are not and some files in the Arma engines cannot be because bis forgot they had this code. The intent behind compression was for internet use and, in the 'good old days', simply to reduce hard disk storage requirements. The actual use of compression is becoming less 'popular' as it does represent a small load on the engine. Operation Flashpoint: Elite cannot work with compressed .pbo files - see Elite PBOs.
PboTypes
Mission Versus Addon
Across all engines there are two distinct types of pbo.
- Mission pbos do not contain config.cpps and live in a mission folder (be it campaign, single, or multiplayer). They have no use for a pboPrefix, nor a Properties header.
- Addon pbos contain a config.cpp/bin which at-the-least identifies the addon name, and, with the exception of the initial cwc engine, rely on a pbo properties header. They live in an addons\ folder.
pbo engine differences
- Cold War Crisis: the very first in the series (disregarding the 'proof of concept' release a year earlier) has no pboProperties header and no checksums. It does however form the core of all other engines. It's format has never been altered
- Resistance: Added a Properties header as the first 'entry' of the pbo.
- Vbs1: same as Resistance (but encrypted).
- Xbox Elite: Added a 0 plus 4 byte checksum to end of file.
- Arma: Changed elite's checksum to a 0 plus 20 byte sha key to end of file.
- VBS2: same as arma (but encrypted)
- Dayz: same as arma
Note also that in almost all cases both the properties header and/or the sha key are optional.
Main format
The format of a .pbo contains:
- a header consisting of 21byte contiguous file name structures called 'entries'. The very first entry might exceed 21 bytes (see below)
- one, contiguous data block.
- an (optional) 5 byte checksum (Elite) or a 21 byte signature key (Arma)
With exceptions, each entry defines a file contained in the .pbo, its size, date, name, it is whether it is compressed.
Because entries and data are contiguous, there is no need for an offset to the 'next' file. Every file, even zero-length ones, are recorded in the header.
However, note that there is no provision for, and no ability to, store empty folders. Folders as such are indicated simply by being part of the filename. There are no, folder entries, and consequently, empty folders, cannot be included in a .pbo because there is no filename associated with them. Put another way, an empty folder, if it could be stored (and it can't), would appear to be an empty filename when dePbo'd.
The last header 'entry' is filled with zeroes. The next byte is the beginning of the data block.
PBO Header Entry
A standard .pbo entry as follows:
struct entry
{
Asciiz filename; // a zero terminated string defining the path and filename,
// ''relative to the name of this .pbo'' or it is prefix.
// Zero length filenames ('\0') indicate the first (optional), or last (non optional) entry in header.
// Other fields in the last entry are filled by zero bytes.
char[4] MimeType; // 0x56657273 'Vers' properties entry (only first entry if at all)
// 0x43707273 'Cprs' compressed entry
// 0x456e6372 'Enco' comressed (vbs)
// 0x00000000 dummy last header entry
ulong OriginalSize; // Uncompressed: 0 or same value as the DataSize
// compressed: Size of file after unpacking.
// This value is needed for byte boundary unpacking
// since unpacking itself can lead to bleeding of up
// to 7 extra bytes.
ulong Offset; // not actually used, always zeros (but vbs = encryption data)
ulong TimeStamp; // meant to be the unix filetime of Jan 1 1970 +, but often 0
ulong DataSize; // The size in the data block.
// This is also the file size when '''not''' packed
};
Null Entries
Entries with no file name indicate:
- End of header, content all zeros and ignored regardless.
- PboProperties entry as the very first entry of the file (not present for cwc or any mission.pbo)
PboProperties
struct standard entry
{
Asciiz filename; // = 0
char[4] MimeType; // 0x56657273 'Vers' properties entry (only first entry if at all)
ulong OriginalSize; // = 0
ulong Reserved; // = 0
ulong TimeStamp; // = 0
ulong DataSize; // = 0
} // end of 'standard' entry
struct properties
{
Asciiz this1, that1; // eg this=that;
}[...];
byte end; // = 0
There can be as many contiguos paired Strings as, well, as many as, a piece of string!
The LAST (or only!) String is a zero length ASCIIZ string (eg. '\0').
Operation Flashpoint: Resistance PBO
struct properties
{
"product" = "OFP: Resistance"
};
Operation Flashpoint: Elite PBO
struct properties
{
"prefix" = "Addon\FOLDER\Name"
};
Arma PBO
prefix=Addon\FOLDER\Name
version="123"
engine="arma3"
author=I am famous"
anything= else that takes your fancy
End of File Checksum or Sha
Operation Flashpoint
Not present for Operation Flashpoint pbos
Operation Flashpoint: Elite
{
byte always0;
int checksum;
}
Arma
{
byte always0;
byte sha[20]; //std md5 checksum only used for MP play
}
Data compression
Data compression is a mild, but effective use of Apple Corps LZSS (a variant of liv-zempel & Huffman), allowing (up to) 8k of previous data to repeat itself.
Compression is indicated when a signature of 0x43707273 and the file sizes do not match in the entry.
The following code also applies to the packing method employed in wrp (OPRW) and pac/paa files which have no header info simply a block of known output length that must be decoded. In all cases, the OUTPUT size is known. With .pbo's, the INPUT size is only a boundary definition to the next block of compressed data. It is not used or relevant to decoding data because (up to) 7 residual bytes could exist in the last flag word of the block. As such, only the fixed in concrete output size is relevant.
The compressed data block is in contiguous 'packets' of different lengths:
block {packet1}...{packetN} {4 byte checksum}
packet
{
byte flagbits;
byte packetdata[...]; // no fixed length
}
The contents of the packetdata contain mixtures of raw data that is passed directly to the output, and, 2byte pointers.
Format: bit values determine what the packetdata is. It is interpeted lsb first thus:
BitN = 1 - append byte directly to file (read single byte) BitN = 0 - pointer (read two bytes)
for example:
format byte, is 0x45, binary notation is: 01000101.
There are three bytes in the block a little further past the format flag that will be passed directly to the output when encountered, and there are FIVE pointers.
In this example, first byte of packetdata is passed to output, 2 bytes are read to make a pointer, next byte is passed (ultimately) to output and so on.
For the very last packet in the block, it is almost inevitable that there will be excessive bits. These are ignored (truncated) as the final output length is always known from the Entry. You cannot rely on the ignored bits in the format flag (up to seven of them) to be any particular value (0 or 1) and must be ignored.
A pointer consists of a 12 bits address and 4 bit run length.
The pointer is a reference to somewhere in the previous 4k max of built output. Given Intel's endian word format the bytes b1 and b2 form a short word value B2B1
The format of B2B1 is unfortunately in Big Endian (motorola) AAAA LLLL AAAAAAAA, requiring some shift and mask fiddling.
The address refers to the start of some data in the currently rebuilt part of the file. It is a value, relative to the current length of the reconstructed part of the file (FL).
The run length of the data to be copied, the 'pattern' has 4 bits and therefore, in theory, 0 to 15 bytes can be duplicated. In practice the values are 3..18 bytes because copying 0,1 or 2 bytes makes no sense.
Relative position (rpos) into the currently built output is calculated as
rpos = FL - ((B2B1 &0x00FF) + (B2B1 & 0xF000)>>4) )
The length of the data block: rlen
rlen = (B2B1 & 0x0F00>>8) + 3
With the values of rpos and rlen there are three basic situations possible:
rpos + rlen < FL // bytes to copy are within the existing reconstructed data block is added to the end of the file, giving a new length of FL = FL + rlen.
rpos + rlen > FL // data to copy exceeds what's available In this situation the data block has a length of FL – rpos and it is added to the reconstructed file until FL = rpos + rlen.
rpos + rlen < 0 This is a special case where spaces are added to the decoded file until FL = FL,Initial + rlen
The checksum, the last four bytes of any compressed data block. It is an unsigned long (Intel Little Endian order).
It is simply a byte-at-a-time, unsigned additive spillover of the decompressed data.
Each and every compressed data block, contains its own, unique checksum.
There is no checksum or other protective device, employed on a .pbo overall. Exceptions: Elite and Arma have residual data after the end of contiguous data block that do represent a signature for the file.
Bibliography
- DosTools: http://dev-heaven.net/projects/list_files/mikero-pbodll
- cpbo: http://www.kegetys.net/arma/
Open Source PBO Libraries
C
- JAPM: https://github.com/RaJiska/JAPM
- armake: https://github.com/KoffeinFlummi/armake
- libpbo: https://github.com/Learath2/libpbo
C++
C#
- SwiftPbo: https://github.com/headswe/SwiftPbo
- PboSharp: https://github.com/Shix/PBOSharp
Python
yapbol: https://github.com/overfl0/yapbol
pbo-fuse: https://github.com/Dahlgren/python-pbo-fuse/blob/master/pbo.py
Java
- ArmaFiles: https://github.com/Krzmbrzl/ArmaFiles