PBO File Format
PBO file structure and packing method
Introduction
A .PBO file means 'packed bank of files'. A .pbo is identical in purpose to a zip or rar. It is a container for folder(s) and file(s).
The engine will internally expand any *.pbo back out to it's original, tree-folder, form.
Legend
see Generic FileFormat Data Types
Compression
In addition to simply packaging all files and folders in a tree into a single file, some, all, or none of the files within can be compressed. Which type of files are compressed is entirely optional. The intent behind compression was for internet use and, in the 'good old days', simply to reduce hard disk storage requirements. The actual use of compression (a mild form of run length encoding) is becoming less 'popular' as it does represent a load on the engine. Operation Flashpoint Elite cannot work with compressed .pbo files. See Elite PBO's
Main format
The format of a .pbo contains:
- a header consisting of 21byte contiguous file name structures called 'entries'. The very first entry might exceed 21 bytes (see below)
- one, contiguous data block.
- an (optional) 5 byte checksum (Elite) or a 21 byte signature key (Arma)
With exceptions, each entry defines a file contained in the .pbo, its size, date, name, it's whether it's compressed.
Because entries and data are contiguous, there is no need for an offset to the 'next' file. Every file, even zero-length ones, are recorded in the header.
However, note that there is no provision for, and no ability to, store empty folders. Folders as such are indicated simply by being part of the filename. There are no, folder entries, and consequently, empty folders, cannot be included in a .pbo because there is no filename associated with them. Put another way, an empty folder, if it could be stored (and it can't), would appear to be an empty filename when dePbo'd.
The last header 'entry' is filled with zeroes. The next byte is the beginning of the data block.
PBO Header Entry
A standard .pbo entry as follows
struct entry
{
Asciiz filename; //a zero terminated string defining the path and filename,
//relative to the name of this .pbo or it's prefix.
//Zero length filenames ('\0') indicate the first (optional), or last (non optional) entry in header.
// Other fields in the last entry are filled by zero bytes.
.
char[4] MimeType; //0x56657273 'Vers' properties entry (only first entry if at all)
//0x43707273 'Cprs' compressed entry
//0x456e6372 'Enco' comressed (vbs)
//0x00000000 dummy last header entry
ulong OriginalSize; // Uncompressed: 0 or same value as the DataSize
// compressed: Size of file after unpacking.
// This value is needed for byte boundary unpacking
// since unpacking itself can lead to bleeding of up
// to 7 extra bytes.
ulong Offset; // not actually used, always zeros (but vbs = encryption data)
ulong TimeStamp; // meant to be the unix filetime of Jan 1 1970 +, but often 0
ulong DataSize; // The size in the data block.
// This is also the file size when not packed
};
Null Entries
entries with no file name indicate
- End of header, content all zeros and ignored regardless.
- PboProperties entry as the very first entry of the file (not present for cwc or any mission.pbo)
PboProperties
struct standard entry
{
Asciiz filename; // = 0
char[4] MimeType; //0x56657273 'Vers' properties entry (only first entry if at all)
ulong OriginalSize; // =0
ulong Reserved;//=0
ulong TimeStamp;//=0
ulong DataSize; //=0
}// end of 'standard' entry
struct properties
{
Asciiz this1,that1;// eg this=that;
}[...];
byte end; //=0
There can be as many contiguos paired Strings as, well, as many as, a piece of string!
The LAST (or only!) String is a zero length Asciiz string (eg. '\0').
Resistance PBO
struct properties
{
"product" = "OFP: Resistance"
};
OFP XBOX Elite PBO
struct properties
{
"prefix" = "Addon\FOLDER\Name"
};
Arma PBO
prefix=Addon\FOLDER\Name version="123" engine="arma3" author=I am famous" anything= else that takes your fancy
End of File Checksum or Sha
flashpoint
Not present for flashpoint pbos
=xbox elite
{
byte always0;
int checksum;
}
Arma
{
byte always0;
byte sha[20]; //std md5 checksum only used for MP play
}
Data compression
Data compression in OFP is a mild, but effective, form of run length encoding (LZH), allowing (up to) 4k of previous data to repeat itself.
Compression is indicated when a signature of 0x43707273 and the filesizes do not match in the entry.
The following code also applies to the packing method employed in wrp (OPRW) and pac/paa files which have no header info simply a block of known output length that must be decoded. In all cases, the OUTPUT size is known. With .pbo's, the INPUT size is only a boundary definition to the next block of compressed data. It is not used or relevant to decoding data because (up to) 7 residual bytes could exist in the last flag word of the block. As such, only the fixed in concrete output size is relevant.
The compressed data block is in contiguous 'packets' of different lengths
block {packet1}...{packetN} {4 byte checksum}
.
packet
{
byte Format;
byte packetdata[...]; // no fixed length
}
The contents of the packetdata contain mixtures of raw data that is passed directly to the output, and, 2byte pointers.
Format: bit values determine what the packetdata is. It is interpeted lsb first thus;
BitN =1 - append byte directly to file (read single byte) BitN= 0 - pointer (read two bytes)
for example:
format byte, is 0x45, binary notation is: 01000101.
There are three bytes in the block a little further past the format flag that will be passed directly to the output when encountered, and there are FIVE pointers.
In this example, first byte of packetdata is passed to output, 2 bytes are read to make a pointer, next byte is passed (ultimately) to output and so on.
For the very last packet in the block, it is almost inevitable that there will be excessive bits. These are ignored (truncated) as the final output length is always known from the Entry. You cannot rely on the ignored bits in the format flag (up to seven of them) to be any particular value (0 or 1).
A pointer consists of a 12 bits address and 4 bit run length.
The pointer is a reference to somewhere in the previous 4k max of built output. Given Intel's endian word format the bytes b1 and b2 form a short word value B2B1
The format of B2B1 is unfortunately AAAA LLLL AAAAAAAA, requiring a bit of shift mask fiddling.
The address refers to the start of some data in the currently rebuilt part of the file. It is a value, relative to the current length of the reconstructed part of the file (FL).
The run length of the data to be copied, the 'pattern' has 4 bits and therefore, in theory, 0 to 15 bytes can be duplicated. In practice the values are 3..18 bytes because copying 0,1 or 2 bytes makes no sense.
Relative position (rpos) into the currently built output is calculated as
rpos = FL - ((B2B1 &0x00FF) + (B2B1 & 0xF000)>>4) )
The length of the data block: rlen
rlen = (B2B1 & 0x0F00>>8) + 3
With the values of rpos and rlen there are three basic situations possible:
rpos + rlen < FL // bytes to copy are within the existing reconstructed data block is added to the end of the file, giving a new length of FL = FL + rlen.
rpos + rlen > FL // data to copy exceeds what's available
In this situation the data block has a length of FL – rpos and it is added to the reconstructed file until FL = rpos + rlen.
rpos + rlen < 0 This is a special case where spaces are added to the decoded file until FL = FL,Initial + rlen
The checksum, the last four bytes of any compressed data block. It is an unsigned long (Intel Little Endian order). It is simply a byte-at-a-time, unsigned additive spillover of the decompressed data.
Each and every compressed data block, contains it's own, unique checksum.
There is no, checksum, or other protective device, employed on a .pbo overall. Exceptions: Elite and Arma have residual data after the end of contiguous data block that do, represent, a signature for the file.
Bibliography
DosTools : http://dev-heaven.net/projects/list_files/mikero-pbodll cpbo : http://www.kegetys.net/arma/
Open Source PBO Libraries
C
JAPM: https://github.com/RaJiska/JAPM
armake: https://github.com/KoffeinFlummi/armake
libpbo: https://github.com/Learath2/libpbo
C#
SwiftPbo: https://github.com/headswe/SwiftPbo
PboSharp: https://github.com/Shix/PBOSharp
C++
libpbo: https://github.com/StidOfficial/libpbo
Python
yapbol: https://github.com/overfl0/yapbol
pbo-fuse: https://github.com/Dahlgren/python-pbo-fuse/blob/master/pbo.py
Java
ArmaFiles: https://github.com/Krzmbrzl/ArmaFiles