嵌入式系统中,存储空间、内存等资源相对较敏感。如何高效的利用有限的系统资源,一直是嵌入式系统设计的重点。在嵌入式linux系统的远程固件(程序)升级功能上尤为突出。如果固件升级包太大,对内存和flash都会带来压力。因此,必须尽量减小固件升级包的大小。
在这个问题上,7z压缩格式为我们带来了福音。相比.tar、.tar.gz、.tar.bz2等压缩格式,采用7z格式压缩的固件升级包明显较小。reille实现的固件升级功能中都采用这种格式制作固件升级包。
关于7z的介绍请参考:7z格式、LZMA压缩算法和7-Zip详细介绍
本文主要分享了移植7z命令到嵌入式linux系统的方法和经验。
有两种7z命令移植到arm-linux平台:一种是完整功能版本的7z命令移植;另一种精简版本的7z命令的移植(只有解压7z格式包功能),下面分别以9.20版本为例进行说明。
1. 完整功能7z命令即7za的移植
1.1 下载7z源码包
在http://sourceforge.net/projects/p7zip/files/或http://sourceforge.net/projects/p7zip/files/p7zip/上下载p7zip的源码包,当前最新版本是9.20.1;
找到对应版本号进去,页面会提供两个供你下载,一个是bin包,另一个是源码包,这里下的是bin包,以9.20.1为例,下载的包名称为:p7zip_9.20.1_src_all.tar.bz2;
1.2 交叉编译和移植7z
解压源码包p7zip_9.20.1_src_all.tar.bz2后,进入源码包,会发现有一个makefile和N多其它平台的makefile:makefile.aix_gcc、makefile.linux_s390x 、makefile.linux_cross_arm等。其中,makefile.linux_cross_arm就是我们要的makefile文件。此外,源码目录中提供了readme文件,其中有说明如下:
According to your OS, copy makefile.linux,
makefile.freebsd, makefile.cygwin, …
over makefile.machine
其中的 makefile.machine 就是平台相关性选项。所以我们可以这样做 :
编辑makefile.linux_cross_arm文件 ,指定其中的编译器路径,如下打红色地方所示:
#
# makefile for Linux (CROSS ARM)
#
OPTFLAGS=-O
ALLFLAGS=${OPTFLAGS} -pipe -s \
-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE \
-DNDEBUG -D_REENTRANT -DENV_UNIX \
-D_7ZIP_LARGE_PAGES \
$(LOCAL_FLAGS)
CXX=/usr/local/arm-uclibc/bin/arm-uclibc-g++ $(ALLFLAGS)
CC=/usr/local/arm-uclibc/bin/arm-uclibc-gcc $(ALLFLAGS)
CC_SHARED=-fPIC -DPIC
LINK_SHARED=-fPIC -DPIC -shared
LOCAL_LIBS=-lpthread
LOCAL_LIBS_DLL=$(LOCAL_LIBS) -ldl
OBJ_CRC32=$(OBJ_CRC32_C)
注:用这个编译器,会有一个错误:提示wine_date_and_time.cpp文件中112行的timegm未定义,可以在config.h的31行,把那个宏定义取消即可正常编译通过。
然后:
$ ln -s makefile.linux_cross_arm makefile.machine
$ make
make后,可以在bin目录下看到有一个7za的文件,这就是我们要的,把其放到arm板上,即可正常运行(关于7z命令的使用,参考:7z命令的使用)。
完整功能版本的7z命令,交叉编译出来的7z命令即7za,其大小为1.38M,不同的编译器可能编译出来的大小有点出入,比如我在公司编译的是1.7M多。
2. 精简功能7z命令即7zDec的移植
精简版7z命令是基于LZMA SDK移植的,关于LZMA SDK,请参考:LZMA SDK介绍
很显然,交叉编译的完整功能7za命令太大,达到了1M多,这对于嵌入式根文件系统来说,一般情况下是不能接收的。但如果从源码包p7zip_9.20.1_src_all.tar.bz2中进行裁减,发现难度很大。
幸好,7z开源项目提供了精简功能7z命令即7zDec。交叉编译后的大小为50KB左右。
1) 下载LZMA-SDK源码包,下载地址:http://www.7-zip.org/sdk.html,当前最新版本是9.20,即lzma920.tar.bz2包;
2) 解压:把源码包的程序文件解压到文件夹:lzma920中;
3) 解压后,建议先查看下其中的说明文件:7zC.txt和lzma.txt两文件;
4) 进入目录:lzma920/C/Util/7z,可以看到其中有两个makefile文件:makefile和makefile.gcc,因为是在linux下,所以用makefile.gcc这个makefile文件;
5) 更改makefile.gcc文件,更改如下:
STRIP = /usr/local/arm-uclibc/bin/arm-uclibc-strip
LIB =
RM = rm -f
CFLAGS = -c -O2 -Wall
OBJS = 7zMain.o 7zAlloc.o 7zBuf.o 7zBuf2.o 7zCrc.o 7zCrcOpt.o 7zDec.o 7zIn.o CpuArch.o LzmaDec.o Lzma2Dec.o Bra.o Bra86.o Bcj2.o Ppmd7.o Ppmd7Dec.o 7zFile.o 7zStream.o
all: $(PROG)
$(PROG): $(OBJS)
$(CXX) -o $(PROG) $(LDFLAGS) $(OBJS) $(LIB)
$(STRIP) $(PROG)
7zMain.o: 7zMain.c
$(CXX) $(CFLAGS) 7zMain.c
7zAlloc.o: ../../7zAlloc.c
$(CXX) $(CFLAGS) ../../7zAlloc.c
7zBuf.o: ../../7zBuf.c
$(CXX) $(CFLAGS) ../../7zBuf.c
7zBuf2.o: ../../7zBuf2.c
$(CXX) $(CFLAGS) ../../7zBuf2.c
7zCrc.o: ../../7zCrc.c
$(CXX) $(CFLAGS) ../../7zCrc.c
7zCrcOpt.o: ../../7zCrc.c
$(CXX) $(CFLAGS) ../../7zCrcOpt.c
7zDec.o: ../../7zDec.c
$(CXX) $(CFLAGS) -D_7ZIP_PPMD_SUPPPORT ../../7zDec.c
7zIn.o: ../../7zIn.c
$(CXX) $(CFLAGS) ../../7zIn.c
CpuArch.o: ../../CpuArch.c
$(CXX) $(CFLAGS) ../../CpuArch.c
LzmaDec.o: ../../LzmaDec.c
$(CXX) $(CFLAGS) ../../LzmaDec.c
Lzma2Dec.o: ../../Lzma2Dec.c
$(CXX) $(CFLAGS) ../../Lzma2Dec.c
Bra.o: ../../Bra.c
$(CXX) $(CFLAGS) ../../Bra.c
Bra86.o: ../../Bra86.c
$(CXX) $(CFLAGS) ../../Bra86.c
Bcj2.o: ../../Bcj2.c
$(CXX) $(CFLAGS) ../../Bcj2.c
Ppmd7.o: ../../Ppmd7.c
$(CXX) $(CFLAGS) ../../Ppmd7.c
Ppmd7Dec.o: ../../Ppmd7Dec.c
$(CXX) $(CFLAGS) ../../Ppmd7Dec.c
7zFile.o: ../../7zFile.c
$(CXX) $(CFLAGS) ../../7zFile.c
7zStream.o: ../../7zStream.c
$(CXX) $(CFLAGS) ../../7zStream.c
clean:
-$(RM) $(PROG) $(OBJS)
6) 编译:make -f makefile.gcc或make -f makefile.gcc clean all;则会在lzma920/C/Util/7z/目录下生成一个名为7zDec的可执行文件,即移植到精简功能版本7z命令。
7) 7zDec命令默认只支持完整7za命令的三个选项,如下所示:
e: Extract files from archive (without using directory names)
l: List contents of archive
t: Test integrity of archive
x: eXtract files with full paths
而不支持”-o”选项,即 指定解压目录 功能,因为在使用过程中有这个需求,因此,对7zDec命令添加了7za命令中的“-o”选项功能。在lzma920/C/Util/7z/目录下,编辑7zMain.c文件,修改main函数如下(主要见by gyr部分代码):
{
CFileInStream archiveStream;
CLookToRead lookStream;
CSzArEx db;
SRes res;
ISzAlloc allocImp;
ISzAlloc allocTempImp;
UInt16 *temp = NULL;
size_t tempSize = 0;
// add start by gyr 2011.11.19
UInt16 *name = NULL;
UInt16 *outPath = NULL;
size_t outPathLen = 0;
// add end by gyr 2011.11.19
printf("\n7z ANSI-C Decoder " MY_VERSION_COPYRIGHT_DATE "\n\n");
if (numargs == 1)
{
printf(
"Usage: 7zDec <command> <archive_name>\n\n"
"<Commands>\n"
" e: Extract files from archive (without using directory names)\n"
" l: List contents of archive\n"
" t: Test integrity of archive\n"
" x: eXtract files with full paths\n"
" -o{Directory}: set Output directory"); // add by gyr 2011.11.19
return 0;
}
if (numargs < 3)
{
PrintError("incorrect command");
return 1;
}
// add start by gyr 2011.11.19
if (numargs == 4 && strstr(args[3], "-o") != NULL)
{
size_t outPathLenTemp = strlen(args[3]);
if (outPathLenTemp <= 2)
{
PrintError("set Output directory Error");
return 1;
}
outPathLen = outPathLenTemp - 2;
if (args[3][outPathLenTemp-1] != '/')
{
outPathLen = outPathLen + 1;
}
outPath = (UInt16 *)SzAlloc(NULL, outPathLen * sizeof(outPath[0]));
if (outPath == NULL)
{
PrintError("can not allocate memory for set output dirctory");
return 1;
}
int j = 0;
for (j=2; j<outPathLenTemp; j++)
{
outPath[j-2] = args[3][j];
}
if (args[3][outPathLenTemp-1] != '/')
{
outPath[outPathLen-1] = '/';
}
}
// add end by gyr 2011.11.19
allocImp.Alloc = SzAlloc;
allocImp.Free = SzFree;
allocTempImp.Alloc = SzAllocTemp;
allocTempImp.Free = SzFreeTemp;
if (InFile_Open(&archiveStream.file, args[2]))
{
PrintError("can not open input file");
return 1;
}
FileInStream_CreateVTable(&archiveStream);
LookToRead_CreateVTable(&lookStream, False);
lookStream.realStream = &archiveStream.s;
LookToRead_Init(&lookStream);
CrcGenerateTable();
SzArEx_Init(&db);
res = SzArEx_Open(&db, &lookStream.s, &allocImp, &allocTempImp);
if (res == SZ_OK)
{
char *command = args[1];
int listCommand = 0, testCommand = 0, extractCommand = 0, fullPaths = 0;
if (strcmp(command, "l") == 0) listCommand = 1;
else if (strcmp(command, "t") == 0) testCommand = 1;
else if (strcmp(command, "e") == 0) extractCommand = 1;
else if (strcmp(command, "x") == 0) { extractCommand = 1; fullPaths = 1; }
else
{
PrintError("incorrect command");
res = SZ_ERROR_FAIL;
}
if (res == SZ_OK)
{
UInt32 i;
/*
if you need cache, use these 3 variables.
if you use external function, you can make these variable as static.
*/
UInt32 blockIndex = 0xFFFFFFFF; /* it can have any value before first call (if outBuffer = 0) */
Byte *outBuffer = 0; /* it must be 0 before first call for each new archive. */
size_t outBufferSize = 0; /* it can have any value before first call (if outBuffer = 0) */
for (i = 0; i < db.db.NumFiles; i++)
{
size_t offset = 0;
size_t outSizeProcessed = 0;
const CSzFileItem *f = db.db.Files + i;
size_t len;
if (listCommand == 0 && f->IsDir && !fullPaths)
continue;
len = SzArEx_GetFileNameUtf16(&db, i, NULL);
if (len > tempSize)
{
SzFree(NULL, temp);
tempSize = len;
temp = (UInt16 *)SzAlloc(NULL, tempSize * sizeof(temp[0]));
if (temp == 0)
{
res = SZ_ERROR_MEM;
break;
}
}
SzArEx_GetFileNameUtf16(&db, i, temp);
if (listCommand)
{
char attr[8], s[32], t[32];
GetAttribString(f->AttribDefined ? f->Attrib : 0, f->IsDir, attr);
UInt64ToStr(f->Size, s);
if (f->MTimeDefined)
ConvertFileTimeToString(&f->MTime, t);
else
{
size_t j;
for (j = 0; j < 19; j++)
t[j] = ' ';
t[j] = '\0';
}
printf("%s %s %10s ", t, attr, s);
res = PrintString(temp);
if (res != SZ_OK)
break;
if (f->IsDir)
printf("/");
printf("\n");
continue;
}
fputs(testCommand ?
"Testing ":
"Extracting ",
stdout);
res = PrintString(temp);
if (res != SZ_OK)
break;
if (f->IsDir)
printf("/");
else
{
res = SzArEx_Extract(&db, &lookStream.s, i,
&blockIndex, &outBuffer, &outBufferSize,
&offset, &outSizeProcessed,
&allocImp, &allocTempImp);
if (res != SZ_OK)
break;
}
if (!testCommand)
{
CSzFile outFile;
size_t processedSize;
size_t j;
// UInt16 *name = (UInt16 *)temp; // del by gyr 2011.11.19
// add start by gyr 2011.11.19
if (name != NULL)
{
SzFree(NULL, name);
}
name = (UInt16 *)SzAlloc(NULL, (tempSize+outPathLen) * sizeof(name[0]));
if (name == NULL)
{
res = SZ_ERROR_MEM;
break;
}
for (j=0; j<outPathLen; j++)
{
name[j] = outPath[j];
}
for (j=outPathLen; j<(tempSize+outPathLen); j++)
{
name[j] = temp[j-outPathLen];
}
// add end by gyr 2011.11.19
const UInt16 *destPath = (const UInt16 *)name;
for (j = 0; name[j] != 0; j++)
if (name[j] == '/')
{
if (fullPaths)
{
name[j] = 0;
MyCreateDir(name);
name[j] = CHAR_PATH_SEPARATOR;
}
else
destPath = name + j + 1;
}
if (f->IsDir)
{
MyCreateDir(destPath);
printf("\n");
continue;
}
else if (OutFile_OpenUtf16(&outFile, destPath))
{
PrintError("can not open output file");
res = SZ_ERROR_FAIL;
break;
}
processedSize = outSizeProcessed;
if (File_Write(&outFile, outBuffer + offset, &processedSize) != 0 || processedSize != outSizeProcessed)
{
PrintError("can not write output file");
res = SZ_ERROR_FAIL;
break;
}
if (File_Close(&outFile))
{
PrintError("can not close output file");
res = SZ_ERROR_FAIL;
break;
}
#ifdef USE_WINDOWS_FILE
if (f->AttribDefined)
SetFileAttributesW(destPath, f->Attrib);
#endif
}
printf("\n");
}
IAlloc_Free(&allocImp, outBuffer);
}
}
SzArEx_Free(&db, &allocImp);
SzFree(NULL, temp);
// add start by gyr 2011.11.19
SzFree(NULL, name);
SzFree(NULL, outPath);
// add end by gyr 2011.11.19
File_Close(&archiveStream.file);
if (res == SZ_OK)
{
printf("\nEverything is Ok\n");
return 0;
}
if (res == SZ_ERROR_UNSUPPORTED)
PrintError("decoder doesn't support this archive");
else if (res == SZ_ERROR_MEM)
PrintError("can not allocate memory");
else if (res == SZ_ERROR_CRC)
PrintError("CRC error");
else
printf("\nERROR #%d\n", res);
return 1;
}
到这里,就基本上完成了7z命令的移植了。请根据需求进行移植。
在我所应用的系统中,一般是tar命令+7zDec命令,在嵌入式linux不进行7z的压缩。