Off-Topic@Heyuri

Anonymous 2025/04/03(Thu)23:33:26 No.137014 [Report] +

This probably needs a python script (thxfully we're in the age of chatgpt)

It should use links.txt 137011-san posted, if a page is redirecting to anywhere on error.fc2.com from the index page, it shouldn't save it and put the link to error.txt with a new line. Successful ones should be written to something like done.txt (will be used to generate the link index if we ever put it on Heyuri etc) after completing. Also it should have a "resuming" mechanism somehow using the two files, checking the last link in both of them & comparing which one is furher down in links.txt, and continuing from the one after that.

After downloading all pages, we should
1- convert downloaded .htm and .html files from SHIFT_JIS to UTF-8
there is a linux command noted on Museum@Heyuri for this: find . -type f \( -name '*.htm' -o -name '*.html' \) -exec sh -c 'iconv -f SHIFT_JIS -t UTF-8//TRANSLIT "{}" > "{}.tmp" && mv "{}.tmp" "{}"' \; or find . -type f \( -name '*.htm' -o -name '*.html' \) -exec sh -c 'iconv -f SHIFT_JIS -t UTF-8 "{}" | sed -r "s/‾/~/g;s/¥/\\\\/g" > "{}.tmp" && mv "{}.tmp" "{}"' \; for saving yen symbols
Or use teh powershell script: https://up.heyuri.net/src/3485.ps1 (Convert-Encoding.ps1)
2- convert <html> at the beginning of each file to <html lang="ja"> so browsers can display intended fonts.
also a command for this noted on Museum@Heyuri: find . -type f \( -name "*.html" -o -name "*.htm" \) -exec sed -i 's/<html\( lang="[^"]*"\)\?>/<html lang="ja">/Ig' {} +
3- remove the end of service notification popup thing
It should detect starting from "<div id="popup-container">" to "</script>" and remove in between
I didn't test but ChatGPT suggested this command: find . -type f \( -iname "*.html" -o -iname "*.htm" \) -exec perl -0777 -i -pe 's|<div id="popup-container">.*?</script>\s*||gs' {} +

Then all that's left is sharing with internets somehow biggrin

Depending how much it actually takes (I doubt the average gets anywhere close to 100MB), we can host it ourselves.

There will be some sites like http://uguisuhp.fc2web.com which doesn't link anything from its index but needs to be linked from http://uguisuhp.fc2web.com/newpage2.htm
I will remember to save this one specifically, but there are probably some others with hidden pages that will get lost...

Preview
Post number	No.
Board
Reason	Optional. Describe what's wrong with it.

Preview	Anonymous 2025/04/03(Thu)23:33:26 No.137014 [Report] + ▶ This probably needs a python script (thxfully we're in the age of chatgpt) It should use links.txt 137011-san posted, if a page is redirecting to anywhere on error.fc2.com from the index page, it shouldn't save it and put the link to error.txt with a new line. Successful ones should be written to something like done.txt (will be used to generate the link index if we ever put it on Heyuri etc) after completing. Also it should have a "resuming" mechanism somehow using the two files, checking the last link in both of them & comparing which one is furher down in links.txt, and continuing from the one after that. After downloading all pages, we should 1- convert downloaded .htm and .html files from SHIFT_JIS to UTF-8 there is a linux command noted on Museum@Heyuri for this: find . -type f \( -name '.htm' -o -name '.html' \) -exec sh -c 'iconv -f SHIFT_JIS -t UTF-8//TRANSLIT "{}" > "{}.tmp" && mv "{}.tmp" "{}"' \; or find . -type f \( -name '.htm' -o -name '.html' \) -exec sh -c 'iconv -f SHIFT_JIS -t UTF-8 "{}" \| sed -r "s/‾/~/g;s/¥/\\\\/g" > "{}.tmp" && mv "{}.tmp" "{}"' \; for saving yen symbols Or use teh powershell script: https://up.heyuri.net/src/3485.ps1 (Convert-Encoding.ps1) 2- convert <html> at the beginning of each file to <html lang="ja"> so browsers can display intended fonts. also a command for this noted on Museum@Heyuri: find . -type f \( -name ".html" -o -name ".htm" \) -exec sed -i 's/<html\( lang="[^"]"\)\?>/<html lang="ja">/Ig' {} +* 3- remove the end of service notification popup thing It should detect starting from "<div id="popup-container">" to "</script>" and remove in between I didn't test but ChatGPT suggested this command: find . -type f \( -iname ".html" -o -iname ".htm" \) -exec perl -0777 -i -pe 's\|<div id="popup-container">.?</script>\s\|\|gs' {} + Then all that's left is sharing with internets somehow Depending how much it actually takes (I doubt the average gets anywhere close to 100MB), we can host it ourselves. There will be some sites like http://uguisuhp.fc2web.com which doesn't link anything from its index but needs to be linked from http://uguisuhp.fc2web.com/newpage2.htm I will remember to save this one specifically, but there are probably some others with hidden pages that will get lost...
Post number	No.137014
Board	Off-Topic@Heyuri
Reason	Optional. Describe what's wrong with it.

Report a post

Heyuri!

Bulletin Boards

Heyuri★CGI

Other

Off-Topic@Heyuri

Report a post