Embed assets in elf file (#2466)

* FBT: file_assets generator
* Elf file: process manifest section externally
* FBT, file_assets generator: add assets signature
* Storage: assets path alias
* Flipper application: assets unpacker
* Apps, Storage: use '/data' alias for apps data
* Storage: copy file to file
* Assets: log flag, fixes
* Update f18 api
* Assets: asserts
* Assets: fix signature_data check
* App assets: example
* Example assets: fix folder structure in readme
* Assets: fix error handling
* Assets builder: use ansii instead of utf-8, use .fapassets section instead of .fapfiles, add assets path to signature
* Elf file: comment strange places
* Storage: totaly optimized storage_file_copy_to_file
This commit is contained in:
Sergey Gavrilov 2023-03-09 18:01:53 +03:00 committed by GitHub
parent 50ef5deefc
commit 4fd043398a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
22 changed files with 937 additions and 113 deletions

View File

@ -362,8 +362,8 @@ static size_t storage_test_apps_count = COUNT_OF(storage_test_apps);
static int32_t storage_test_app(void* arg) {
UNUSED(arg);
Storage* storage = furi_record_open(RECORD_STORAGE);
storage_common_remove(storage, "/app/test");
int32_t ret = storage_file_create(storage, "/app/test", "test");
storage_common_remove(storage, "/data/test");
int32_t ret = storage_file_create(storage, "/data/test", "test");
furi_record_close(RECORD_STORAGE);
return ret;
}
@ -401,7 +401,7 @@ MU_TEST(test_storage_data_path) {
Storage* storage = furi_record_open(RECORD_STORAGE);
File* file = storage_file_alloc(storage);
mu_check(storage_dir_open(file, "/app"));
mu_check(storage_dir_open(file, "/data"));
mu_check(storage_dir_close(file));
storage_file_free(file);

View File

@ -0,0 +1,58 @@
# Apps Assets folder Example
This example shows how to use the Apps Assets folder to store data that is not part of the application itself, but is required for its operation, and that data is provided with the application.
## What is the Apps Assets Folder?
The **Apps Assets** folder is a folder where external applications unpack their assets.
The path to the current application folder is related to the `appid` of the app. The `appid` is used to identify the app in the app store and is stored in the `application.fam` file.
The Apps Assets folder is located only on the external storage, the SD card.
For example, if the `appid` of the app is `snake_game`, the path to the Apps Assets folder will be `/ext/apps_assets/snake_game`. But using raw paths is not recommended, because the path to the Apps Assets folder can change in the future. Use the `/assets` alias instead.
## How to get the path to the Apps Assets folder?
You can use `/assets` alias to get the path to the current application data folder. For example, if you want to open a file `database.txt` in the Apps Assets folder, you can use the next path: `/data/database.txt`. But this way is not recommended, because even the `/assets` alias can change in the future.
We recommend to use the `APP_ASSETS_PATH` macro to get the path to the Apps Assets folder. For example, if you want to open a file `database.txt` in the Apps Assets folder, you can use the next path: `APP_ASSETS_PATH("database.txt")`.
## What is the difference between the Apps Assets folder and the Apps Data folder?
The Apps Assets folder is used to store the data <u>provided</u> with the application. For example, if you want to create a game, you can store game levels (contant data) in the Apps Assets folder.
The Apps Data folder is used to store data <u>generated</u> by the application. For example, if you want to create a game, you can save the progress of the game (user-generated data) in the Apps Data folder.
## How to provide the data with the app?
To provide data with an application, you need to create a folder inside your application folder (eg "files") and place the data in it. After that, you need to add `fap_file_assets="files"` to your application.fam file.
For example, if you want to provide game levels with the application, you need to create a "levels" folder inside the "files" folder and put the game levels in it. After that, you need to add `fap_file_assets="files"` to your application.fam file. The final application folder structure will look like this:
```
snake_game
├── application.fam
├── snake_game.c
└── files
└── levels
├── level1.txt
├── level2.txt
└── level3.txt
```
When app is launched, the `files` folder will be unpacked to the Apps Assets folder. The final structure of the Apps Assets folder will look like this:
```
/assets
├── .assets.signature
└── levels
├── level1.txt
├── level2.txt
└── level3.txt
```
## When will the data be unpacked?
The data is unpacked when the application starts, if the application is launched for the first time, or if the data within the application is updated.
When an application is compiled, the contents of the "files" folder are hashed and stored within the application itself. When the application starts, this hash is compared to the hash stored in the `.assets.signature` file. If the hashes differ or the `.assets.signature` file does not exist, the application folder is deleted and the new data is unpacked.

View File

@ -0,0 +1,10 @@
App(
appid="example_apps_assets",
name="Example: Apps Assets",
apptype=FlipperAppType.EXTERNAL,
entry_point="example_apps_assets_main",
requires=["gui"],
stack_size=4 * 1024,
fap_category="Examples",
fap_file_assets="files",
)

View File

@ -0,0 +1,48 @@
#include <furi.h>
#include <storage/storage.h>
#include <toolbox/stream/stream.h>
#include <toolbox/stream/file_stream.h>
// Define log tag
#define TAG "example_apps_assets"
static void example_apps_data_print_file_content(Storage* storage, const char* path) {
Stream* stream = file_stream_alloc(storage);
FuriString* line = furi_string_alloc();
FURI_LOG_I(TAG, "----------------------------------------");
FURI_LOG_I(TAG, "File \"%s\" content:", path);
if(file_stream_open(stream, path, FSAM_READ, FSOM_OPEN_EXISTING)) {
while(stream_read_line(stream, line)) {
furi_string_replace_all(line, "\r", "");
furi_string_replace_all(line, "\n", "");
FURI_LOG_I(TAG, "%s", furi_string_get_cstr(line));
}
} else {
FURI_LOG_E(TAG, "Failed to open file");
}
FURI_LOG_I(TAG, "----------------------------------------");
furi_string_free(line);
file_stream_close(stream);
stream_free(stream);
}
// Application entry point
int32_t example_apps_assets_main(void* p) {
// Mark argument as unused
UNUSED(p);
// Open storage
Storage* storage = furi_record_open(RECORD_STORAGE);
example_apps_data_print_file_content(storage, APP_ASSETS_PATH("test_asset.txt"));
example_apps_data_print_file_content(storage, APP_ASSETS_PATH("poems/a jelly-fish.txt"));
example_apps_data_print_file_content(storage, APP_ASSETS_PATH("poems/theme in yellow.txt"));
example_apps_data_print_file_content(storage, APP_ASSETS_PATH("poems/my shadow.txt"));
// Close storage
furi_record_close(RECORD_STORAGE);
return 0;
}

View File

@ -0,0 +1,24 @@
A Jelly-Fish by Marianne Moore
Visible, invisible,
A fluctuating charm,
An amber-colored amethyst
Inhabits it; your arm
Approaches, and
It opens and
It closes;
You have meant
To catch it,
And it shrivels;
You abandon
Your intent—
It opens, and it
Closes and you
Reach for it—
The blue
Surrounding it
Grows cloudy, and
It floats away
From you.
source: "https://poets.org/anthology/poems-your-poetry-project-public-domain"

View File

@ -0,0 +1,23 @@
My Shadow by Robert Louis Stevenson
I have a little shadow that goes in and out with me,
And what can be the use of him is more than I can see.
He is very, very like me from the heels up to the head;
And I see him jump before me, when I jump into my bed.
The funniest thing about him is the way he likes to grow—
Not at all like proper children, which is always very slow;
For he sometimes shoots up taller like an India-rubber ball,
And he sometimes gets so little that theres none of him at all.
He hasnt got a notion of how children ought to play,
And can only make a fool of me in every sort of way.
He stays so close beside me, hes a coward you can see;
Id think shame to stick to nursie as that shadow sticks to me!
One morning, very early, before the sun was up,
I rose and found the shining dew on every buttercup;
But my lazy little shadow, like an arrant sleepy-head,
Had stayed at home behind me and was fast asleep in bed.
source: "https://poets.org/anthology/poems-your-poetry-project-public-domain"

View File

@ -0,0 +1,19 @@
Theme in Yellow by Carl Sandburg
I spot the hills
With yellow balls in autumn.
I light the prairie cornfields
Orange and tawny gold clusters
And I am called pumpkins.
On the last of October
When dusk is fallen
Children join hands
And circle round me
Singing ghost songs
And love to the harvest moon;
I am a jack-o'-lantern
With terrible teeth
And the children know
I am fooling.
source: "https://poets.org/anthology/poems-your-poetry-project-public-domain"

View File

@ -0,0 +1 @@
## This is test file content

View File

@ -9,10 +9,16 @@ The **Apps Data** folder is a folder used to store data for external apps that a
The path to the current application folder is related to the `appid` of the app. The `appid` is used to identify the app in the app store and is stored in the `application.fam` file.
The Apps Data folder is located only on the external storage, the SD card.
For example, if the `appid` of the app is `snake_game`, the path to the Apps Data folder will be `/ext/apps_data/snake_game`. But using raw paths is not recommended, because the path to the Apps Data folder can change in the future. Use the `/app` alias instead.
For example, if the `appid` of the app is `snake_game`, the path to the Apps Data folder will be `/ext/apps_data/snake_game`. But using raw paths is not recommended, because the path to the Apps Data folder can change in the future. Use the `/data` alias instead.
## How to get the path to the Apps Data folder?
You can use `/app` alias to get the path to the current application data folder. For example, if you want to open a file `config.txt` in the Apps Data folder, you can use the next path: `/app/config.txt`. But this way is not recommended, because even the `/app` alias can change in the future.
You can use `/data` alias to get the path to the current application data folder. For example, if you want to open a file `config.txt` in the Apps Data folder, you can use the next path: `/data/config.txt`. But this way is not recommended, because even the `/data` alias can change in the future.
We recommend to use the `APP_DATA_PATH` macro to get the path to the Apps Data folder. For example, if you want to open a file `config.txt` in the Apps Data folder, you can use the next path: `APP_DATA_PATH("config.txt")`.
We recommend to use the `APP_DATA_PATH` macro to get the path to the Apps Data folder. For example, if you want to open a file `config.txt` in the Apps Data folder, you can use the next path: `APP_DATA_PATH("config.txt")`.
## What is the difference between the Apps Assets folder and the Apps Data folder?
The Apps Assets folder is used to store the data <u>provided</u> with the application. For example, if you want to create a game, you can store game levels (contant data) in the Apps Assets folder.
The Apps Data folder is used to store data <u>generated</u> by the application. For example, if you want to create a game, you can save the progress of the game (user-generated data) in the Apps Data folder.

View File

@ -10,12 +10,14 @@ extern "C" {
#define STORAGE_INT_PATH_PREFIX "/int"
#define STORAGE_EXT_PATH_PREFIX "/ext"
#define STORAGE_ANY_PATH_PREFIX "/any"
#define STORAGE_APP_DATA_PATH_PREFIX "/app"
#define STORAGE_APP_DATA_PATH_PREFIX "/data"
#define STORAGE_APP_ASSETS_PATH_PREFIX "/assets"
#define INT_PATH(path) STORAGE_INT_PATH_PREFIX "/" path
#define EXT_PATH(path) STORAGE_EXT_PATH_PREFIX "/" path
#define ANY_PATH(path) STORAGE_ANY_PATH_PREFIX "/" path
#define APP_DATA_PATH(path) STORAGE_APP_DATA_PATH_PREFIX "/" path
#define APP_ASSETS_PATH(path) STORAGE_APP_ASSETS_PATH_PREFIX "/" path
#define RECORD_STORAGE "storage"
@ -146,6 +148,17 @@ bool storage_file_eof(File* file);
*/
bool storage_file_exists(Storage* storage, const char* path);
/**
* @brief Copy data from one opened file to another opened file
* Size bytes will be copied from current position of source file to current position of destination file
*
* @param source source file
* @param destination destination file
* @param size size of data to copy
* @return bool success flag
*/
bool storage_file_copy_to_file(File* source, File* destination, uint32_t size);
/******************* Dir Functions *******************/
/** Opens a directory to get objects from it

View File

@ -9,6 +9,7 @@
#define MAX_NAME_LENGTH 256
#define MAX_EXT_LEN 16
#define FILE_BUFFER_SIZE 512
#define TAG "StorageAPI"
@ -251,6 +252,26 @@ bool storage_file_exists(Storage* storage, const char* path) {
return exist;
}
bool storage_file_copy_to_file(File* source, File* destination, uint32_t size) {
uint8_t* buffer = malloc(FILE_BUFFER_SIZE);
while(size) {
uint32_t read_size = size > FILE_BUFFER_SIZE ? FILE_BUFFER_SIZE : size;
if(storage_file_read(source, buffer, read_size) != read_size) {
break;
}
if(storage_file_write(destination, buffer, read_size) != read_size) {
break;
}
size -= read_size;
}
free(buffer);
return size == 0;
}
/****************** DIR ******************/
static bool storage_dir_open_internal(File* file, const char* path) {

View File

@ -13,6 +13,7 @@ extern "C" {
#define STORAGE_COUNT (ST_INT + 1)
#define APPS_DATA_PATH EXT_PATH("apps_data")
#define APPS_ASSETS_PATH EXT_PATH("apps_assets")
typedef struct {
ViewPort* view_port;

View File

@ -454,7 +454,7 @@ void storage_process_alias(
FuriString* apps_data_path_with_appsid = furi_string_alloc_set(APPS_DATA_PATH "/");
furi_string_cat(apps_data_path_with_appsid, furi_thread_get_appid(thread_id));
// "/app" -> "/ext/apps_data/appsid"
// "/data" -> "/ext/apps_data/appsid"
furi_string_replace_at(
path,
0,
@ -472,6 +472,18 @@ void storage_process_alias(
}
furi_string_free(apps_data_path_with_appsid);
} else if(furi_string_start_with(path, STORAGE_APP_ASSETS_PATH_PREFIX)) {
FuriString* apps_assets_path_with_appsid = furi_string_alloc_set(APPS_ASSETS_PATH "/");
furi_string_cat(apps_assets_path_with_appsid, furi_thread_get_appid(thread_id));
// "/assets" -> "/ext/apps_assets/appsid"
furi_string_replace_at(
path,
0,
strlen(STORAGE_APP_ASSETS_PATH_PREFIX),
furi_string_get_cstr(apps_assets_path_with_appsid));
furi_string_free(apps_assets_path_with_appsid);
}
}

View File

@ -1,5 +1,5 @@
entry,status,name,type,params
Version,+,18.0,,
Version,+,18.1,,
Header,+,applications/services/bt/bt_service/bt.h,,
Header,+,applications/services/cli/cli.h,,
Header,+,applications/services/cli/cli_vcp.h,,
@ -1636,6 +1636,7 @@ Function,-,storage_dir_rewind,_Bool,File*
Function,+,storage_error_get_desc,const char*,FS_Error
Function,+,storage_file_alloc,File*,Storage*
Function,+,storage_file_close,_Bool,File*
Function,+,storage_file_copy_to_file,_Bool,"File*, File*, uint32_t"
Function,+,storage_file_eof,_Bool,File*
Function,+,storage_file_exists,_Bool,"Storage*, const char*"
Function,+,storage_file_free,void,File*

1 entry status name type params
2 Version + 18.0 18.1
3 Header + applications/services/bt/bt_service/bt.h
4 Header + applications/services/cli/cli.h
5 Header + applications/services/cli/cli_vcp.h
1636 Function + storage_error_get_desc const char* FS_Error
1637 Function + storage_file_alloc File* Storage*
1638 Function + storage_file_close _Bool File*
1639 Function + storage_file_copy_to_file _Bool File*, File*, uint32_t
1640 Function + storage_file_eof _Bool File*
1641 Function + storage_file_exists _Bool Storage*, const char*
1642 Function + storage_file_free void File*

View File

@ -1,5 +1,5 @@
entry,status,name,type,params
Version,+,18.0,,
Version,+,18.1,,
Header,+,applications/services/bt/bt_service/bt.h,,
Header,+,applications/services/cli/cli.h,,
Header,+,applications/services/cli/cli_vcp.h,,
@ -2461,6 +2461,7 @@ Function,-,storage_dir_rewind,_Bool,File*
Function,+,storage_error_get_desc,const char*,FS_Error
Function,+,storage_file_alloc,File*,Storage*
Function,+,storage_file_close,_Bool,File*
Function,+,storage_file_copy_to_file,_Bool,"File*, File*, uint32_t"
Function,+,storage_file_eof,_Bool,File*
Function,+,storage_file_exists,_Bool,"Storage*, const char*"
Function,+,storage_file_free,void,File*

1 entry status name type params
2 Version + 18.0 18.1
3 Header + applications/services/bt/bt_service/bt.h
4 Header + applications/services/cli/cli.h
5 Header + applications/services/cli/cli_vcp.h
2461 Function + storage_error_get_desc const char* FS_Error
2462 Function + storage_file_alloc File* Storage*
2463 Function + storage_file_close _Bool File*
2464 Function + storage_file_copy_to_file _Bool File*, File*, uint32_t
2465 Function + storage_file_eof _Bool File*
2466 Function + storage_file_exists _Bool Storage*, const char*
2467 Function + storage_file_free void File*

View File

@ -0,0 +1,361 @@
#include "application_assets.h"
#include <toolbox/path.h>
#include <storage/storage_i.h>
// #define ELF_ASSETS_DEBUG_LOG 1
#ifndef ELF_ASSETS_DEBUG_LOG
#undef FURI_LOG_D
#define FURI_LOG_D(...)
#undef FURI_LOG_E
#define FURI_LOG_E(...)
#endif
#define FLIPPER_APPLICATION_ASSETS_MAGIC 0x4F4C5A44
#define FLIPPER_APPLICATION_ASSETS_VERSION 1
#define FLIPPER_APPLICATION_ASSETS_SIGNATURE_FILENAME ".assets.signature"
#define BUFFER_SIZE 512
#define TAG "fap_assets"
#pragma pack(push, 1)
typedef struct {
uint32_t magic;
uint32_t version;
uint32_t dirs_count;
uint32_t files_count;
} FlipperApplicationAssetsHeader;
#pragma pack(pop)
typedef enum {
AssetsSignatureResultEqual,
AssetsSignatureResultNotEqual,
AssetsSignatureResultError,
} AssetsSignatureResult;
static FuriString* flipper_application_assets_alloc_app_full_path(FuriString* app_name) {
furi_assert(app_name);
FuriString* full_path = furi_string_alloc_set(APPS_ASSETS_PATH "/");
furi_string_cat(full_path, app_name);
return full_path;
}
static FuriString* flipper_application_assets_alloc_signature_file_path(FuriString* app_name) {
furi_assert(app_name);
FuriString* signature_file_path = flipper_application_assets_alloc_app_full_path(app_name);
furi_string_cat(signature_file_path, "/" FLIPPER_APPLICATION_ASSETS_SIGNATURE_FILENAME);
return signature_file_path;
}
static uint8_t* flipper_application_assets_alloc_and_load_data(File* file, size_t* size) {
furi_assert(file);
uint8_t* data = NULL;
uint32_t length = 0;
// read data length
if(storage_file_read(file, &length, sizeof(length)) != sizeof(length)) {
return NULL;
}
data = malloc(length);
// read data
if(storage_file_read(file, (void*)data, length) != length) {
free((void*)data);
return NULL;
}
if(size != NULL) {
*size = length;
}
return data;
}
static bool flipper_application_assets_process_files(
Storage* storage,
File* file,
FuriString* app_name,
uint32_t files_count) {
furi_assert(storage);
furi_assert(file);
furi_assert(app_name);
UNUSED(storage);
bool success = false;
uint32_t length = 0;
char* path = NULL;
FuriString* file_path = furi_string_alloc();
File* destination = storage_file_alloc(storage);
FuriString* full_path = flipper_application_assets_alloc_app_full_path(app_name);
for(uint32_t i = 0; i < files_count; i++) {
path = (char*)flipper_application_assets_alloc_and_load_data(file, NULL);
if(path == NULL) {
break;
}
// read file size
if(storage_file_read(file, &length, sizeof(length)) != sizeof(length)) {
break;
}
furi_string_set(file_path, full_path);
furi_string_cat(file_path, "/");
furi_string_cat(file_path, path);
if(!storage_file_open(
destination, furi_string_get_cstr(file_path), FSAM_WRITE, FSOM_CREATE_ALWAYS)) {
FURI_LOG_E(TAG, "Can't create file: %s", furi_string_get_cstr(file_path));
break;
}
// copy data to file
if(!storage_file_copy_to_file(file, destination, length)) {
FURI_LOG_E(TAG, "Can't copy data to file: %s", furi_string_get_cstr(file_path));
break;
}
storage_file_close(destination);
free(path);
path = NULL;
if(i == files_count - 1) {
success = true;
}
}
if(path != NULL) {
free(path);
}
storage_file_free(destination);
furi_string_free(file_path);
return success;
}
static bool flipper_application_assets_process_dirs(
Storage* storage,
File* file,
FuriString* app_name,
uint32_t dirs_count) {
furi_assert(storage);
furi_assert(file);
furi_assert(app_name);
bool success = false;
FuriString* full_path = flipper_application_assets_alloc_app_full_path(app_name);
do {
if(!storage_simply_mkdir(storage, APPS_ASSETS_PATH)) {
break;
}
if(!storage_simply_mkdir(storage, furi_string_get_cstr(full_path))) {
break;
}
FuriString* dir_path = furi_string_alloc();
char* path = NULL;
for(uint32_t i = 0; i < dirs_count; i++) {
path = (char*)flipper_application_assets_alloc_and_load_data(file, NULL);
if(path == NULL) {
break;
}
furi_string_set(dir_path, full_path);
furi_string_cat(dir_path, "/");
furi_string_cat(dir_path, path);
if(!storage_simply_mkdir(storage, furi_string_get_cstr(dir_path))) {
FURI_LOG_E(TAG, "Can't create directory: %s", furi_string_get_cstr(dir_path));
break;
}
free(path);
path = NULL;
if(i == dirs_count - 1) {
success = true;
}
}
if(path != NULL) {
free(path);
}
furi_string_free(dir_path);
} while(false);
furi_string_free(full_path);
return success;
}
static AssetsSignatureResult flipper_application_assets_process_signature(
Storage* storage,
File* file,
FuriString* app_name,
uint8_t** signature_data,
size_t* signature_data_size) {
furi_assert(storage);
furi_assert(file);
furi_assert(app_name);
furi_assert(signature_data);
furi_assert(signature_data_size);
AssetsSignatureResult result = AssetsSignatureResultError;
File* signature_file = storage_file_alloc(storage);
FuriString* signature_file_path =
flipper_application_assets_alloc_signature_file_path(app_name);
do {
// read signature
*signature_data =
flipper_application_assets_alloc_and_load_data(file, signature_data_size);
if(*signature_data == NULL) { //-V547
FURI_LOG_E(TAG, "Can't read signature");
break;
}
result = AssetsSignatureResultNotEqual;
if(!storage_file_open(
signature_file,
furi_string_get_cstr(signature_file_path),
FSAM_READ_WRITE,
FSOM_OPEN_EXISTING)) {
FURI_LOG_E(TAG, "Can't open signature file");
break;
}
size_t signature_size = storage_file_size(signature_file);
uint8_t* signature_file_data = malloc(signature_size);
if(storage_file_read(signature_file, signature_file_data, signature_size) !=
signature_size) {
FURI_LOG_E(TAG, "Can't read signature file");
free(signature_file_data);
break;
}
if(memcmp(*signature_data, signature_file_data, signature_size) == 0) {
FURI_LOG_D(TAG, "Assets signature is equal");
result = AssetsSignatureResultEqual;
}
free(signature_file_data);
} while(0);
storage_file_free(signature_file);
furi_string_free(signature_file_path);
return result;
}
bool flipper_application_assets_load(File* file, const char* elf_path, size_t offset, size_t size) {
UNUSED(size);
furi_assert(file);
furi_assert(elf_path);
FlipperApplicationAssetsHeader header;
bool result = false;
Storage* storage = furi_record_open(RECORD_STORAGE);
uint8_t* signature_data = NULL;
size_t signature_data_size = 0;
FuriString* app_name = furi_string_alloc();
path_extract_filename_no_ext(elf_path, app_name);
FURI_LOG_D(TAG, "Loading assets for %s", furi_string_get_cstr(app_name));
do {
if(!storage_file_seek(file, offset, true)) {
break;
}
// read header
if(storage_file_read(file, &header, sizeof(header)) != sizeof(header)) {
break;
}
if(header.magic != FLIPPER_APPLICATION_ASSETS_MAGIC) {
break;
}
if(header.version != FLIPPER_APPLICATION_ASSETS_VERSION) {
break;
}
// process signature
AssetsSignatureResult signature_result = flipper_application_assets_process_signature(
storage, file, app_name, &signature_data, &signature_data_size);
if(signature_result == AssetsSignatureResultError) {
FURI_LOG_E(TAG, "Assets signature error");
break;
} else if(signature_result == AssetsSignatureResultEqual) {
FURI_LOG_D(TAG, "Assets signature equal, skip loading");
result = true;
break;
} else {
FURI_LOG_D(TAG, "Assets signature not equal, loading");
// remove old assets
FuriString* full_path = flipper_application_assets_alloc_app_full_path(app_name);
storage_simply_remove_recursive(storage, furi_string_get_cstr(full_path));
furi_string_free(full_path);
FURI_LOG_D(TAG, "Assets removed");
}
// process directories
if(!flipper_application_assets_process_dirs(storage, file, app_name, header.dirs_count)) {
break;
}
// process files
if(!flipper_application_assets_process_files(storage, file, app_name, header.files_count)) {
break;
}
// write signature
FuriString* signature_file_path =
flipper_application_assets_alloc_signature_file_path(app_name);
File* signature_file = storage_file_alloc(storage);
if(storage_file_open(
signature_file,
furi_string_get_cstr(signature_file_path),
FSAM_WRITE,
FSOM_CREATE_ALWAYS)) {
storage_file_write(signature_file, signature_data, signature_data_size);
}
storage_file_free(signature_file);
furi_string_free(signature_file_path);
result = true;
} while(false);
if(signature_data != NULL) {
free(signature_data);
}
furi_record_close(RECORD_STORAGE);
furi_string_free(app_name);
FURI_LOG_D(TAG, "Assets loading %s", result ? "success" : "failed");
return result;
}

View File

@ -0,0 +1,17 @@
/**
* @file application_assets.h
* Flipper application assets
*/
#pragma once
#include <storage/storage.h>
#ifdef __cplusplus
extern "C" {
#endif
bool flipper_application_assets_load(File* file, const char* elf_path, size_t offset, size_t size);
#ifdef __cplusplus
}
#endif

View File

@ -241,7 +241,7 @@ static void elf_relocate_jmp_call(ELFFile* elf, Elf32_Addr relAddr, int type, El
if(to_thumb || (symAddr & 2) || (!is_call)) {
FURI_LOG_D(
TAG,
"can't relocate value at %x, %s, doing trampoline",
"can't relocate value at %lx, %s, doing trampoline",
relAddr,
elf_reloc_type_to_str(type));
@ -421,29 +421,11 @@ typedef enum {
SectionTypeRelData = 1 << 2,
SectionTypeSymTab = 1 << 3,
SectionTypeStrTab = 1 << 4,
SectionTypeManifest = 1 << 5,
SectionTypeDebugLink = 1 << 6,
SectionTypeDebugLink = 1 << 5,
SectionTypeValid = SectionTypeSymTab | SectionTypeStrTab | SectionTypeManifest,
SectionTypeValid = SectionTypeSymTab | SectionTypeStrTab,
} SectionType;
static bool elf_load_metadata(
ELFFile* elf,
Elf32_Shdr* section_header,
FlipperApplicationManifest* manifest) {
if(section_header->sh_size < sizeof(FlipperApplicationManifest)) {
return false;
}
if(manifest == NULL) {
return true;
}
return storage_file_seek(elf->fd, section_header->sh_offset, true) &&
storage_file_read(elf->fd, manifest, section_header->sh_size) ==
section_header->sh_size;
}
static bool elf_load_debug_link(ELFFile* elf, Elf32_Shdr* section_header) {
elf->debug_link_info.debug_link_size = section_header->sh_size;
elf->debug_link_info.debug_link = malloc(section_header->sh_size);
@ -478,7 +460,7 @@ static bool elf_load_section_data(ELFFile* elf, ELFSection* section, Elf32_Shdr*
return false;
}
FURI_LOG_D(TAG, "0x%X", section->data);
FURI_LOG_D(TAG, "0x%p", section->data);
return true;
}
@ -486,8 +468,7 @@ static SectionType elf_preload_section(
ELFFile* elf,
size_t section_idx,
Elf32_Shdr* section_header,
FuriString* name_string,
FlipperApplicationManifest* manifest) {
FuriString* name_string) {
const char* name = furi_string_get_cstr(name_string);
#ifdef ELF_DEBUG_LOG
@ -572,16 +553,6 @@ static SectionType elf_preload_section(
return SectionTypeStrTab;
}
// Load manifest section
if(strcmp(name, ".fapmeta") == 0) {
FURI_LOG_D(TAG, "Found .fapmeta section");
if(elf_load_metadata(elf, section_header, manifest)) {
return SectionTypeManifest;
} else {
return SectionTypeERROR;
}
}
// Load debug link section
if(strcmp(name, ".gnu_debuglink") == 0) {
FURI_LOG_D(TAG, "Found .gnu_debuglink section");
@ -692,41 +663,12 @@ bool elf_file_open(ELFFile* elf, const char* path) {
return true;
}
bool elf_file_load_manifest(ELFFile* elf, FlipperApplicationManifest* manifest) {
bool result = false;
FuriString* name;
name = furi_string_alloc();
FURI_LOG_D(TAG, "Looking for manifest section");
for(size_t section_idx = 1; section_idx < elf->sections_count; section_idx++) {
Elf32_Shdr section_header;
furi_string_reset(name);
if(!elf_read_section(elf, section_idx, &section_header, name)) {
break;
}
if(furi_string_cmp(name, ".fapmeta") == 0) {
if(elf_load_metadata(elf, &section_header, manifest)) {
FURI_LOG_D(TAG, "Load manifest done");
result = true;
break;
} else {
break;
}
}
}
furi_string_free(name);
return result;
}
bool elf_file_load_section_table(ELFFile* elf, FlipperApplicationManifest* manifest) {
bool elf_file_load_section_table(ELFFile* elf) {
SectionType loaded_sections = SectionTypeERROR;
FuriString* name;
name = furi_string_alloc();
FuriString* name = furi_string_alloc();
FURI_LOG_D(TAG, "Scan ELF indexs...");
// TODO: why we start from 1?
for(size_t section_idx = 1; section_idx < elf->sections_count; section_idx++) {
Elf32_Shdr section_header;
@ -738,8 +680,7 @@ bool elf_file_load_section_table(ELFFile* elf, FlipperApplicationManifest* manif
FURI_LOG_D(
TAG, "Preloading data for section #%d %s", section_idx, furi_string_get_cstr(name));
SectionType section_type =
elf_preload_section(elf, section_idx, &section_header, name, manifest);
SectionType section_type = elf_preload_section(elf, section_idx, &section_header, name);
loaded_sections |= section_type;
if(section_type == SectionTypeERROR) {
@ -753,14 +694,49 @@ bool elf_file_load_section_table(ELFFile* elf, FlipperApplicationManifest* manif
return IS_FLAGS_SET(loaded_sections, SectionTypeValid);
}
ElfProcessSectionResult elf_process_section(
ELFFile* elf,
const char* name,
ElfProcessSection* process_section,
void* context) {
ElfProcessSectionResult result = ElfProcessSectionResultNotFound;
FuriString* section_name = furi_string_alloc();
Elf32_Shdr section_header;
// find section
// TODO: why we start from 1?
for(size_t section_idx = 1; section_idx < elf->sections_count; section_idx++) {
furi_string_reset(section_name);
if(!elf_read_section(elf, section_idx, &section_header, section_name)) {
break;
}
if(furi_string_cmp(section_name, name) == 0) {
result = ElfProcessSectionResultCannotProcess;
break;
}
}
if(result != ElfProcessSectionResultNotFound) { //-V547
if(process_section(elf->fd, section_header.sh_offset, section_header.sh_size, context)) {
result = ElfProcessSectionResultSuccess;
} else {
result = ElfProcessSectionResultCannotProcess;
}
}
furi_string_free(section_name);
return result;
}
ELFFileLoadStatus elf_file_load_sections(ELFFile* elf) {
ELFFileLoadStatus status = ELFFileLoadStatusSuccess;
ELFSectionDict_it_t it;
AddressCache_init(elf->relocation_cache);
for(ELFSectionDict_it(it, elf->sections); !ELFSectionDict_end_p(it);
ELFSectionDict_next(it)) {
for(ELFSectionDict_it(it, elf->sections); !ELFSectionDict_end_p(it); ELFSectionDict_next(it)) {
ELFSectionDict_itref_t* itref = ELFSectionDict_ref(it);
FURI_LOG_D(TAG, "Relocating section '%s'", itref->key);
if(!elf_relocate_section(elf, &itref->value)) {

View File

@ -37,6 +37,14 @@ typedef enum {
ELFFileLoadStatusMissingImports,
} ELFFileLoadStatus;
typedef enum {
ElfProcessSectionResultNotFound,
ElfProcessSectionResultCannotProcess,
ElfProcessSectionResultSuccess,
} ElfProcessSectionResult;
typedef bool(ElfProcessSection)(File* file, size_t offset, size_t size, void* context);
/**
* @brief Allocate ELFFile instance
* @param storage
@ -59,21 +67,12 @@ void elf_file_free(ELFFile* elf_file);
*/
bool elf_file_open(ELFFile* elf_file, const char* path);
/**
* @brief Load ELF file manifest
* @param elf
* @param manifest
* @return bool
*/
bool elf_file_load_manifest(ELFFile* elf, FlipperApplicationManifest* manifest);
/**
* @brief Load ELF file section table (load stage #1)
* @param elf_file
* @param manifest
* @return bool
*/
bool elf_file_load_section_table(ELFFile* elf_file, FlipperApplicationManifest* manifest);
bool elf_file_load_section_table(ELFFile* elf_file);
/**
* @brief Load and relocate ELF file sections (load stage #2)
@ -122,6 +121,21 @@ void elf_file_init_debug_info(ELFFile* elf_file, ELFDebugInfo* debug_info);
*/
void elf_file_clear_debug_info(ELFDebugInfo* debug_info);
/**
* @brief Process ELF file section
*
* @param elf_file
* @param name
* @param process_section
* @param context
* @return ElfProcessSectionResult
*/
ElfProcessSectionResult elf_process_section(
ELFFile* elf_file,
const char* name,
ElfProcessSection* process_section,
void* context);
#ifdef __cplusplus
}
#endif

View File

@ -1,6 +1,7 @@
#include "flipper_application.h"
#include "elf/elf_file.h"
#include <notification/notification_messages.h>
#include "application_assets.h"
#define TAG "fapp"
@ -55,24 +56,83 @@ static FlipperApplicationPreloadStatus
return FlipperApplicationPreloadStatusSuccess;
}
/* Parse headers, load manifest */
FlipperApplicationPreloadStatus
flipper_application_preload_manifest(FlipperApplication* app, const char* path) {
if(!elf_file_open(app->elf, path) || !elf_file_load_manifest(app->elf, &app->manifest)) {
static bool flipper_application_process_manifest_section(
File* file,
size_t offset,
size_t size,
void* context) {
FlipperApplicationManifest* manifest = context;
if(size < sizeof(FlipperApplicationManifest)) {
return false;
}
if(manifest == NULL) {
return true;
}
return storage_file_seek(file, offset, true) &&
storage_file_read(file, manifest, size) == size;
}
// we can't use const char* as context because we will lose the const qualifier
typedef struct {
const char* path;
} FlipperApplicationPreloadAssetsContext;
static bool flipper_application_process_assets_section(
File* file,
size_t offset,
size_t size,
void* context) {
FlipperApplicationPreloadAssetsContext* preload_context = context;
return flipper_application_assets_load(file, preload_context->path, offset, size);
}
static FlipperApplicationPreloadStatus
flipper_application_load(FlipperApplication* app, const char* path, bool load_full) {
if(!elf_file_open(app->elf, path)) {
return FlipperApplicationPreloadStatusInvalidFile;
}
// if we are loading full file
if(load_full) {
// load section table
if(!elf_file_load_section_table(app->elf)) {
return FlipperApplicationPreloadStatusInvalidFile;
}
// load assets section
FlipperApplicationPreloadAssetsContext preload_context = {.path = path};
if(elf_process_section(
app->elf,
".fapassets",
flipper_application_process_assets_section,
&preload_context) == ElfProcessSectionResultCannotProcess) {
return FlipperApplicationPreloadStatusInvalidFile;
}
}
// load manifest section
if(elf_process_section(
app->elf, ".fapmeta", flipper_application_process_manifest_section, &app->manifest) !=
ElfProcessSectionResultSuccess) {
return FlipperApplicationPreloadStatusInvalidFile;
}
return flipper_application_validate_manifest(app);
}
/* Parse headers, load manifest */
FlipperApplicationPreloadStatus
flipper_application_preload_manifest(FlipperApplication* app, const char* path) {
return flipper_application_load(app, path, false);
}
/* Parse headers, load full file */
FlipperApplicationPreloadStatus
flipper_application_preload(FlipperApplication* app, const char* path) {
if(!elf_file_open(app->elf, path) || !elf_file_load_section_table(app->elf, &app->manifest)) {
return FlipperApplicationPreloadStatusInvalidFile;
}
return flipper_application_validate_manifest(app);
return flipper_application_load(app, path, true);
}
const FlipperApplicationManifest* flipper_application_get_manifest(FlipperApplication* app) {

View File

@ -67,6 +67,7 @@ class FlipperApplication:
fap_icon_assets_symbol: Optional[str] = None
fap_extbuild: List[ExternallyBuiltFile] = field(default_factory=list)
fap_private_libs: List[Library] = field(default_factory=list)
fap_file_assets: Optional[str] = None
# Internally used by fbt
_appdir: Optional[object] = None
_apppath: Optional[str] = None

View File

@ -1,5 +1,5 @@
from dataclasses import dataclass, field
from typing import Optional
from typing import Optional, TypedDict
from SCons.Builder import Builder
from SCons.Action import Action
from SCons.Errors import UserError
@ -15,6 +15,8 @@ import os
import pathlib
import itertools
import shutil
import struct
import hashlib
from ansi.color import fg
@ -151,12 +153,24 @@ def BuildAppElf(env, app):
app_artifacts.compact,
[app_env["SDK_DEFINITION"], app_env.Value(manifest_vals)],
)
# Add dependencies on icon files
if app.fap_icon:
app_env.Depends(
app_artifacts.compact,
app_env.File(f"{app._apppath}/{app.fap_icon}"),
)
# Add dependencies on file assets
if app.fap_file_assets:
app_env.Depends(
app_artifacts.compact,
app_env.GlobRecursive(
"*",
app._appdir.Dir(app.fap_file_assets),
),
)
app_artifacts.validator = app_env.ValidateAppImports(app_artifacts.compact)
app_env.AlwaysBuild(app_artifacts.validator)
app_env.Alias(app_alias, app_artifacts.validator)
@ -266,6 +280,159 @@ def resources_fap_dist_action(target, source, env):
shutil.copy(src.path, target.path)
def generate_embed_app_metadata_emitter(target, source, env):
app = env["APP"]
meta_file_name = source[0].path + ".meta"
target.append("#" + meta_file_name)
if app.fap_file_assets:
files_section = source[0].path + ".files.section"
target.append("#" + files_section)
return (target, source)
class File(TypedDict):
path: str
size: int
content_path: str
class Dir(TypedDict):
path: str
def prepare_app_files(target, source, env):
app = env["APP"]
directory = app._appdir.Dir(app.fap_file_assets)
directory_path = directory.abspath
if not directory.exists():
raise UserError(f"File asset directory {directory} does not exist")
file_list: list[File] = []
directory_list: list[Dir] = []
for root, dirs, files in os.walk(directory_path):
for file_info in files:
file_path = os.path.join(root, file_info)
file_size = os.path.getsize(file_path)
file_list.append(
{
"path": os.path.relpath(file_path, directory_path),
"size": file_size,
"content_path": file_path,
}
)
for dir_info in dirs:
dir_path = os.path.join(root, dir_info)
dir_size = sum(
os.path.getsize(os.path.join(dir_path, f)) for f in os.listdir(dir_path)
)
directory_list.append(
{
"path": os.path.relpath(dir_path, directory_path),
}
)
file_list.sort(key=lambda f: f["path"])
directory_list.sort(key=lambda d: d["path"])
files_section = source[0].path + ".files.section"
with open(files_section, "wb") as f:
# u32 magic
# u32 version
# u32 dirs_count
# u32 files_count
# u32 signature_size
# u8[] signature
# Dirs:
# u32 dir_name length
# u8[] dir_name
# Files:
# u32 file_name length
# u8[] file_name
# u32 file_content_size
# u8[] file_content
# Write header magic and version
f.write(struct.pack("<II", 0x4F4C5A44, 0x01))
# Write dirs count
f.write(struct.pack("<I", len(directory_list)))
# Write files count
f.write(struct.pack("<I", len(file_list)))
md5_hash = hashlib.md5()
md5_hash_size = len(md5_hash.digest())
# write signature size and null signature, we'll fill it in later
f.write(struct.pack("<I", md5_hash_size))
signature_offset = f.tell()
f.write(b"\x00" * md5_hash_size)
# Write dirs
for dir_info in directory_list:
f.write(struct.pack("<I", len(dir_info["path"]) + 1))
f.write(dir_info["path"].encode("ascii") + b"\x00")
md5_hash.update(dir_info["path"].encode("ascii") + b"\x00")
# Write files
for file_info in file_list:
f.write(struct.pack("<I", len(file_info["path"]) + 1))
f.write(file_info["path"].encode("ascii") + b"\x00")
f.write(struct.pack("<I", file_info["size"]))
md5_hash.update(file_info["path"].encode("ascii") + b"\x00")
with open(file_info["content_path"], "rb") as content_file:
content = content_file.read()
f.write(content)
md5_hash.update(content)
# Write signature
f.seek(signature_offset)
f.write(md5_hash.digest())
def generate_embed_app_metadata_actions(source, target, env, for_signature):
app = env["APP"]
actions = [
Action(prepare_app_metadata, "$APPMETA_COMSTR"),
]
objcopy_str = (
"${OBJCOPY} "
"--remove-section .ARM.attributes "
"--add-section .fapmeta=${SOURCE}.meta "
)
if app.fap_file_assets:
actions.append(Action(prepare_app_files, "$APPFILE_COMSTR"))
objcopy_str += "--add-section .fapassets=${SOURCE}.files.section "
objcopy_str += (
"--set-section-flags .fapmeta=contents,noload,readonly,data "
"--strip-debug --strip-unneeded "
"--add-gnu-debuglink=${SOURCE} "
"${SOURCES} ${TARGET}"
)
actions.append(
Action(
objcopy_str,
"$APPMETAEMBED_COMSTR",
)
)
return Action(actions)
def generate(env, **kw):
env.SetDefault(
EXT_APPS_WORK_DIR="${FBT_FAP_DEBUG_ELF_ROOT}",
@ -275,6 +442,7 @@ def generate(env, **kw):
env.SetDefault(
FAPDISTCOMSTR="\tFAPDIST\t${TARGET}",
APPMETA_COMSTR="\tAPPMETA\t${TARGET}",
APPFILE_COMSTR="\tAPPFILE\t${TARGET}",
APPMETAEMBED_COMSTR="\tFAP\t${TARGET}",
APPCHECK_COMSTR="\tAPPCHK\t${SOURCE}",
)
@ -295,21 +463,10 @@ def generate(env, **kw):
emitter=resources_fap_dist_emitter,
),
"EmbedAppMetadata": Builder(
action=[
Action(prepare_app_metadata, "$APPMETA_COMSTR"),
Action(
"${OBJCOPY} "
"--remove-section .ARM.attributes "
"--add-section .fapmeta=${SOURCE}.meta "
"--set-section-flags .fapmeta=contents,noload,readonly,data "
"--strip-debug --strip-unneeded "
"--add-gnu-debuglink=${SOURCE} "
"${SOURCES} ${TARGET}",
"$APPMETAEMBED_COMSTR",
),
],
generator=generate_embed_app_metadata_actions,
suffix=".fap",
src_suffix=".elf",
emitter=generate_embed_app_metadata_emitter,
),
"ValidateAppImports": Builder(
action=[