View Issue Details

Category
SSPBT:本体(SSP)
SeverityfeatureReproducibilityN/A 
Status closed 
Fixed in Version2.5 (リリース) 
Summary0000215: Speed up the construction of update.txt by caching the last modification time of files and folders
Description

If the modification time of a folder is the same as that of the last update.txt build, we can skip them directly, which is very useful for ghost with tens of thousands of files
We can save these caches in "update.cache" or some file name.

Activities

ponapalt

2021-07-20 16:54

administrator   ~0000474

Since update.txt is generated infrequently, I would like to avoid implementing such bug-prone features.

guest

2021-07-29 16:37

reporter   ~0000477

I don't think this feature is easy to cause bugs
But it's up to you to decide. I'll describe the code implementation I think of first. If you don't want to complete it, it's not too late to close this report

  1. Delete the update.cache when the update.txt or update.dau file does not exist
  2. Read update.txt or update.dau and take its contents as std::map< filename, {MD5, size} >
  3. Read the update.cache and take its contents as std::map< filename, modification time >, std::map< dirname, vector<filename> contentitems >
  4. Define update::txtbuilder_mapper(string FileOrDirName,time_t lastchangtime)
  5. Function content: skip this item when the modification date not changed. Otherwise, then like the dicdir implementation in YAYA: if it is a folder, traverse the file or folder under the folder and call it recursively (and query whether the content item recorded in the cache has been deleted, if so, remove the corresponding record in maps). If it is a file, calculate the MD5, Finally, both files and folders update the map content (MD5 and modification date, etc.).
  6. Save the map content to the update files
  7. If build updates.txt and updates.dau at the same time, copy and paste the file

guest

2021-07-29 16:41

reporter   ~0000478

By the way, Do you know https://copilot.github.com/ ? This software can generate code according to your comments.
Although you need to check the logic of the code, it can improve your work efficiency in general

guest

2021-07-31 14:04

reporter   ~0000480

My idea is that if you are worried that the new build method is easy to cause bugs, you can build another entry for the new build method and keep the old build method until the new build method is stable

ponapalt

2021-08-25 15:09

administrator   ~0000594

2.5.17 implemented partially

last modification time of folder isn't always updated when containing files are updated, so I've implemented file-by-file update time check, and skipping hash function when update time is same.

Issue History

Date Modified Username Field Change
2021-07-18 22:54 guest New Issue
2021-07-20 16:54 ponapalt Assigned To => ponapalt
2021-07-20 16:54 ponapalt Status new => closed
2021-07-20 16:54 ponapalt Resolution open => not fixable
2021-07-20 16:54 ponapalt Note Added: 0000474
2021-07-29 16:37 guest Status closed => feedback
2021-07-29 16:37 guest Resolution not fixable => reopened
2021-07-29 16:37 guest Note Added: 0000477
2021-07-29 16:41 guest Note Added: 0000478
2021-07-29 16:41 guest Status feedback => assigned
2021-07-31 14:04 guest Note Added: 0000480
2021-08-23 08:14 ponapalt Status assigned => confirmed
2021-08-25 15:09 ponapalt Status confirmed => closed
2021-08-25 15:09 ponapalt Fixed in Version => 2.5 (リリース)
2021-08-25 15:09 ponapalt Note Added: 0000594