Commit graph

239 commits

Author SHA1 Message Date
woxtu
cdac3988b5
fix(backend): Fix typos in job configurations (#13086)
* Fix typos

* Update CHANGELOG
2024-01-28 15:08:45 +09:00
Marie
913dd581ef
merge: upstream 2024-01-25 14:21:42 +01:00
syuilo
65557d5f27 enhance(reversi): more robust matching process 2024-01-24 10:16:05 +09:00
Marie
7552cea69a
merge: upstream 2024-01-09 02:57:57 +01:00
Kagami Sascha Rosylight
2a9db983fc
feat: export clips (#12931)
* feat: export clips

* Update CHANGELOG.md
2024-01-07 10:35:58 +09:00
riku6460
24645e3d3d
enhance(backend): ActivityPub 周りで連合先から HTTP 429 Too Many Requests を受け取った際にジョブをリトライするように (#12917)
* enhance(backend): ActivityPub 周りで HTTP 429 Too Many Requests を受け取った際にリトライするように

* add to changelog

---------

Co-authored-by: syuilo <Syuilotan@yahoo.co.jp>
2024-01-06 09:40:08 +09:00
MeiMei
d415fd29a3
enhance(backend): ActivityPub Deliver queueでBodyを事前処理するように (#12916)
* Pre-processing deliver body

* CHANGELOG

* ループ内で計算されると意味がないので

* 同じ処理を同じ形に

---------

Co-authored-by: まっちゃとーにゅ <17376330+u1-liquid@users.noreply.github.com>
2024-01-06 09:07:48 +09:00
Marie
1805150533 fix: visibility check on masto import
Originally from PR #288
2023-12-31 22:41:35 +01:00
Marie
233eff48f3 merge: pleroma note import - Use hashed filename for exists check (#283)
Reviewed-on: https://git.joinsharkey.org/Sharkey/Sharkey/pulls/283
2023-12-31 18:43:38 +01:00
smitten
8d6d5923da
Simplify hash steps 2023-12-31 11:14:41 -05:00
smitten
327694d4cf
Use base64url digest 2023-12-31 09:13:51 -05:00
smitten
e9428a5a05
Use hex digest 2023-12-31 09:03:46 -05:00
Marie
b700fadbe3 upd: add home as a visibility for mastodon imports 2023-12-31 06:32:39 +01:00
Marie
07f06d7ed6 fix: if condition 2023-12-31 04:09:44 +01:00
Marie
fc6581b948 fix: correct followers visibility on import 2023-12-31 03:50:05 +01:00
Marie
667daebb79 upd: prevent vanilla mastodon imports from importing DMs
Also adds the visibility function to mastodon imports
2023-12-31 03:48:51 +01:00
smitten
0bb0d69543
Use hashed filename for exists check 2023-12-30 20:44:31 -05:00
Marie
5db583a3eb merge: upstream 2023-12-23 02:09:23 +01:00
Kagami Sascha Rosylight
4175b7809b
chore(QueueProcessorService): show error stack for failures (#12727) 2023-12-21 10:29:30 +09:00
Camilla Ett
06ca63f9c2
Fix(backend): inboxJobPerSecのデフォルト値を16から32に (#12631) 2023-12-13 08:14:34 +09:00
dakkar
c958d935e4 thread Pleroma imports as well
I have _not_ tested this, but it should work fine, those exports are
the same shape as Mastodon's
2023-11-30 13:26:55 +00:00
dakkar
c59e74dfd5 fix chaining for Mastodon notes
the id / replyId are not at the top level, so now `recreateChain`
takes a list of keys to walk, not just a single key
2023-11-30 13:26:55 +00:00
dakkar
15503b96a0 properly thread Mastodon imports 2023-11-30 13:26:55 +00:00
Gianni Ceccarelli
e392d523a7 prepare to import more notes
`recreateChain` converts a list of notes into a forest of notes, using
notes that are not replies as roots, and replies as child nodes,
recursively.

Previously, notes that are replies to notes not included in the
export, and their children, were never put in the forest, and
therefore wheren't imported.

This can be fine when importing from Twitter, since we can't really
link a note to a tweet.

And, for the moment, it's acceptable when importing from *key, because
the export doesn't contain the instance URL, so we can't resolve ids
to remote notes.

It's less fine when importing from Mastodon / Pleroma / Akkoma,
because in those cases we _can_ link to the remote note that the user
was replying to.

This commit makes `recreateChain` optionally return "orphaned" note
trees, so in the (near) future we can use it to properly thread
imported notes from those services.
2023-11-30 13:26:55 +00:00
Mar0xy
62bcd42891
upd: create Folders for imported media 2023-11-28 22:46:10 +01:00
Mar0xy
81c36908d9
fix: attachments not working on FB import 2023-11-22 19:02:43 +01:00
Mar0xy
d078a72f38
upd: add Facebook to note importing 2023-11-22 16:00:46 +01:00
Mar0xy
19fc5a74f0
merge: improve recreateChain
Resolves #159
2023-11-18 12:38:13 +01:00
Mar0xy
4929467016
upd: import own replies to own tweets 2023-11-17 20:10:16 +01:00
Mar0xy
e4bd74183f
chore: remove type in DbNoteImportToDbJobData 2023-11-13 13:09:21 +01:00
Mar0xy
6f76a3a1f8
upd: import replies to own posts on *key 2023-11-13 13:07:49 +01:00
Mar0xy
9b3993a652
fix: prevent *key renotes/boosts from being imported
Stop renotes/boosts from *key from being imported due to missing functionality of getting the related posts
2023-11-13 10:25:39 +01:00
Mar0xy
5c38e6b824
add: custom like emoji per instance, fix like
This fixes the fact that likes on mastodon didn't get federated properly and let's instance admins choose a custom emoji
2023-11-12 22:16:47 +01:00
Mar0xy
1f8c12b984
fix: use a proper function 2023-11-12 18:19:44 +01:00
Mar0xy
8cad8f0ae5
fix: broken unicode in strings on IG imports 2023-11-12 18:12:39 +01:00
Mar0xy
486a2ad839
upd: remove unused function, fix logger message 2023-11-12 18:01:39 +01:00
Mar0xy
bc25b28a4e
fix: cleanup not available on development env 2023-11-12 17:32:24 +01:00
Mar0xy
b7fc13efb7
upd: pass full thinuser 2023-11-12 15:48:28 +01:00
Mar0xy
2d1e8ee44d
upd: update limit 2023-11-12 15:37:36 +01:00
Mar0xy
de9ea4d9f1
fix: type 2023-11-12 15:16:27 +01:00
Mar0xy
83f328de8a add: Importing of Posts
- Supports Instagram, Mastodon/Pleroma/Akkoma, Twitter and *key
2023-11-12 15:12:42 +01:00
Mar0xy
d85cf658ca
upd: export headers used during signin 2023-11-07 18:18:04 +01:00
Mar0xy
42ecde475e
upd: add IPs to data export 2023-11-07 12:20:46 +01:00
Mar0xy
fabb876256
upd: send email on export completion 2023-11-07 11:31:11 +01:00
Mar0xy
e9b6ed941b
add: endpoint and processor for account data export 2023-11-07 03:31:47 +01:00
Mar0xy
33eab0d178
upd: change handling of renewkeyFailed 2023-10-20 12:54:25 +02:00
Mar0xy
c76f0f1679
fix: set boolean to true if uri doesn't match activity actor 2023-10-20 12:53:14 +02:00
Mar0xy
71b7c31958
upd: refetch user keys on signature failure
Reference: https://github.com/misskey-dev/misskey/pull/12051
2023-10-20 12:50:56 +02:00
Mar0xy
4b230d792c
merge: upstream changes 2023-10-18 19:48:03 +02:00
woxtu
0bddd0ceae
Remove unnecessary nullish coalescing (#12058) 2023-10-18 09:54:18 +09:00
anatawa12
5a3c6575dd
新規にフォローした人のwithRepliesをtrueにする機能を追加 (#12048)
* feat: add defaultWithReplies to MiUser

* feat: use defaultWithReplies when creating MiFollowing

* feat: update defaultWithReplies from API

* feat: return defaultWithReplies as a part of $i

* feat(frontend): configure defaultWithReplies

* docs(changelog): 新規にフォローした人のをデフォルトでTL二追加できるように

* fix: typo

* style: fix lint failure

* chore: improve UI text

* chore: make optional params of  UserFollowingService.follow() object

* chore: UserFollowingService.follow() accept withReplies

* chore: add withReplies to MiFollowRequest

* chore: process withReplies for follow request

* feat: accept withReplies on 'following/create' endpoint

* feat: store defaultWithReplies in client store

* Revert "feat: return defaultWithReplies as a part of $i"

This reverts commit f2cc4fe6

* Revert "feat: update defaultWithReplies from API"

This reverts commit 95e3cee6

* Revert "feat: add defaultWithReplies to MiUser"

This reverts commit 9f5ab14d7063532de2b049bc2ed40a15658168f5.

* feat: configuring withReplies in import-following

* feat(frontend): configure withReplies

* fix(frontend): incorrectly showRepliesToOthersInTimeline can be shown

* fix(backend): withReplies of following/create not working

* fix(frontend): importFollowing error

* fix: withReplies is not working with follow import

* fix(frontend): use v-model

* style: fix lint

---------

Co-authored-by: Sayamame-beans <61457993+sayamame-beans@users.noreply.github.com>
Co-authored-by: syuilo <syuilotan@yahoo.co.jp>
2023-10-17 20:56:17 +09:00
Mar0xy
68886b9ab5
merge: upstream performance changes & instance silencing 2023-10-16 23:38:21 +02:00
syuilo
3ebed5aa3e feat: local-only antenna
Resolve #11869
2023-10-16 15:06:00 +09:00
syuilo
1fa1d31696 perf(backend): createdAtをidから取得するように & 無駄なDateインスタンスの生成を避けるように 2023-10-16 10:45:22 +09:00
syuilo
f6ba5cfaf4
merge: timeline 1 2023-10-13 17:58:11 +02:00
Mar0xy
fe4345d191
upd: delete reactions properly in the DB
Fixes https://github.com/misskey-dev/misskey/issues/11906
2023-10-11 23:03:01 +02:00
Mar0xy
bf3d493d04
Revert "feat: improve tl performance" 2023-10-03 20:21:26 +02:00
syuilo
6277a5545c
feat: improve tl performance (#11946)
* wip

* wip

* wip

* wip

* wip

* wip

* Update NoteCreateService.ts

* wip

* wip

* wip

* wip

* Update NoteCreateService.ts

* wip

* Update NoteCreateService.ts

* wip

* Update user-notes.ts

* wip

* wip

* wip

* Update NoteCreateService.ts

* wip

* Update timeline.ts

* Update timeline.ts

* Update timeline.ts

* Update timeline.ts

* Update timeline.ts

* wip

* Update timelines.ts

* Update timelines.ts

* Update timelines.ts

* wip

* wip

* wip

* Update timelines.ts

* Update misskey-js.api.md

* Update timelines.ts

* Update timelines.ts

* wip

* wip

* wip

* Update timelines.ts

* wip

* Update timelines.ts

* wip

* test

* Update activitypub.ts

* refactor: UserListJoining -> UserListMembership

* Update NoteCreateService.ts

* wip
2023-10-03 20:26:11 +09:00
syuilo
b9da1415a5 feat: 通知の受信設定を強化 2023-09-29 11:29:54 +09:00
syuilo
053da10e94 refactor(backend): update directory structure for models 2023-09-20 11:33:36 +09:00
syuilo
6cf466e5d1
update deps (#11820)
* update deps

* fix

* wip

* wip

* wip

* Update docker-compose.yml.example

* Delete reviewer-lottery.yml

* Update RepositoryModule.ts

* wip

* wip

* clean up

* update deps

* wip

* wip
2023-09-15 14:28:29 +09:00
syuilo
d8dc10829c enhance(backend): 古いアンテナを自動停止するか切り替え可能に
Resolve #11785
2023-09-07 16:20:28 +09:00
anatawa12
50ec129b87
feat: 'server' webhook payload which hold misskey server url (#11752) 2023-08-21 16:13:10 +09:00
syuilo
792622aead
refactor: prefix Mi for all entities (#11719)
* wip

* wip

* wip

* wip

* Update RepositoryModule.ts

* wip

* wip

* wip

* Revert "wip"

This reverts commit c1c13b37d2aaf3c65bc148212da302b0eb7868bf.
2023-08-16 17:51:28 +09:00
woxtu
8a6791da3f
refactor(backend): Remove unused injections (#11462)
* Remove unused injections

* Remove unused imports
2023-08-05 10:33:00 +09:00
Shun Sakai
c2370a1be6
chore: 著作権とライセンスについての情報を各ファイルに追加する (#11348)
* chore: Add the SPDX information to each file

Add copyright and licensing information as defined in version 3.0 of
the REUSE Specification.

* tweak format

---------

Co-authored-by: syuilo <Syuilotan@yahoo.co.jp>
2023-07-27 14:31:52 +09:00
tamaina
0a06eb27da
enhance(backend): 設定ファイルでioredisの全てのオプションを指定可能に (#11329)
* enhance(backend): 設定ファイルでioredisの全てのオプションを指定可能に

* yappa kousuru

* fix

* fix?

* fix

* Revert "fix"

This reverts commit 227f19ff3afcbbd560b831493975206263a1a5a3.

* fix
2023-07-20 19:50:31 +09:00
Acid Chicken (硫酸鶏)
e6fca72171
perf: use slacc instead of unzipper (#10780)
Co-authored-by: tamaina <tamaina@hotmail.co.jp>
2023-07-20 17:00:54 +09:00
woxtu
8c77153c04
Fix auto-fixable issues (#11314) 2023-07-19 11:27:50 +09:00
syuilo
ad5fd626c7 fix of #11293 2023-07-16 14:46:41 +09:00
CGsama
4417412787
Export notes with file detail (#11293) 2023-07-16 14:21:49 +09:00
okayurisotto
2b6dbd4fcb
refactor: 可読性のため一部でArray.prototype.atを使うように (#11274)
* refactor: `Array.prototype.at`を使うように

* fixup! refactor: `Array.prototype.at`を使うように
2023-07-14 10:45:01 +09:00
Caipira
60366a4558
fix(backend): Remove Meilisearch index when notes are deleted (#10988)
* fix(backend): Include feature to delete Meilisearch index notes

* Update variable name
`cascadingNotesFilter` -> `federatedLocalCascadingNotes`

* tweak

---------

Co-authored-by: syuilo <Syuilotan@yahoo.co.jp>
2023-07-08 21:31:38 +09:00
okayurisotto
d84796588c
cleanup: trim trailing whitespace (#11136)
* cleanup: trim trailing whitespace

* update(`.editorconfig`)

---------

Co-authored-by: syuilo <Syuilotan@yahoo.co.jp>
2023-07-08 07:08:16 +09:00
syuilo
8099bc24e1
refactor(backend): use node16 for moduleResolution (#10938)
* refactor(backend): use node16 for moduleResolution

* update deps

* Update tsconfig.json

* ✌️

* revive KEYWORD

* restore strict-event-emitter-types dependency

* restore ms dependency

* cancel redundant import reorder

* fix

* Delete ms.ts

* remove rndstr

---------

Co-authored-by: Kagami Sascha Rosylight <saschanaz@outlook.com>
2023-06-25 21:13:15 +09:00
syuilo
b35b9bc27f Update QueueProcessorService.ts 2023-05-29 13:30:57 +09:00
syuilo
4790ddfad6 refactor of QueueProcessorService 2023-05-29 13:30:26 +09:00
syuilo
fd7b77c542
enhance(backend): migrate bull to bullmq (#10910)
* wip

* wip

* Update QueueService.ts

* wip

* refactor

* ✌️

* fix

* Update QueueStatsService.ts

* refactor

* Update ApNoteService.ts

* Update mock-resolver.ts

* refactor

* Update mock-resolver.ts
2023-05-29 11:54:49 +09:00
syuilo
7ce569424a feat: カスタム絵文字ごとにそれをリアクションとして使えるロールを設定できるように 2023-05-18 18:45:49 +09:00
syuilo
341c42ebb9 enhance(backend): graceful shutdown for job queue and refactor 2023-05-10 15:05:08 +09:00
syuilo
c7d4264ecf refactor 2023-05-06 09:34:21 +09:00
Chocolate Pie
39748ea0c3
feat: アンテナのエクスポート・インポート (#10754)
* feat: アンテナのエクスポートに対応 (misskey-dev/misskey#10690)

* feat: アンテナのインポートに対応 (misskey-dev/misskey#10690)

* fix: タイポを修正

* feat: ユーザーリストをサポート

* fix: バグを直した

* fix: バグを直した

* fix: 適当に決めた変数名を変更

* fix

* fix: 変数の変更、リファクタリング
2023-05-06 08:17:55 +09:00
Namekuji
d28866f71a
enhance: account migration (#10592)
* copy block and mute then create follow and unfollow jobs

* copy block and mute and update lists when detecting an account has moved

* no need to care promise orders

* refactor updating actor and target

* automatically accept if a locked account had accepted an old account

* fix exception format

* prevent the old account from calling some endpoints

* do not unfollow when moving

* adjust following and follower counts

* check movedToUri when receiving a follow request

* skip if no need to adjust

* Revert "disable account migration"

This reverts commit 2321214c98.

* fix translation specifier

* fix checking alsoKnownAs and uri

* fix updating account

* fix refollowing locked account

* decrease followersCount if followed by the old account

* adjust following and followers counts when unfollowing

* fix copying mutings

* prohibit moved account from moving again

* fix move service

* allow app creation after moving

* fix lint

* remove unnecessary field

* fix cache update

* add e2e test

* add e2e test of accepting the new account automatically

* force follow if any error happens

* remove unnecessary joins

* use Array.map instead of for const of

* ユーザーリストの移行は追加のみを行う

* nanka iroiro

* fix misskey-js?

* ✌️

* 移行を行ったアカウントからのフォローリクエストの自動許可を調整

* newUriを外に出す

* newUriを外に出す2

* clean up

* fix newUri

* prevent moving if the destination account has already moved

* set alsoKnownAs via /i/update

* fix database initialization

* add return type

* prohibit updating alsoKnownAs after moving

* skip to add to alsoKnownAs if toUrl is known

* skip adding to the list if it already has

* use Acct.parse instead

* rename error code

* 🎨

* 制限を5から10に緩和

* movedTo(Uri), alsoKnownAsはユーザーidを返すように

* test api res

* fix

* 元アカウントはミュートし続ける

* 🎨

* unfollow

* fix

* getUserUriをUserEntityServiceに

* ?

* job!

* 🎨

* instance => server

* accountMovedShort, forbiddenBecauseYouAreMigrated

* accountMovedShort

* fix test

* import, pin禁止

* 実績を凍結する

* clean up

* ✌️

* change message

* ブロック, フォロー, ミュート, リストのインポートファイルの制限を32MiBに

* Revert "ブロック, フォロー, ミュート, リストのインポートファイルの制限を32MiBに"

This reverts commit 3bd7be35d8aa455cb01ae58f8172a71a50485db1.

* validateAlsoKnownAs

* 移行後2時間以内はインポート可能なファイルサイズを拡大

* clean up

* どうせactorをupdatePersonで更新するならupdatePersonしか移行処理を発行しないことにする

* handle error?

* リモートからの移行処理の条件を是正

* log, port

* fix

* fix

* enhance(dev): non-production環境でhttpサーバー間でもユーザー、ノートの連合が可能なように

* refactor (use checkHttps)

* MISSKEY_WEBFINGER_USE_HTTP

* Environment Variable readme

* NEVER USE IN PRODUCTION

* fix punyHost

* fix indent

* fix

* experimental

---------

Co-authored-by: tamaina <tamaina@hotmail.co.jp>
Co-authored-by: syuilo <Syuilotan@yahoo.co.jp>
2023-04-30 00:09:29 +09:00
syuilo
34492f3c9a enhance(backend): tweak cache of federated instance
#10631
2023-04-22 19:59:08 +09:00
tamaina
168fe0e376
fix(server): Force the extention of exported file (#10630)
* fix(server): Force the extention of exported file

* update changelog.md
2023-04-14 14:35:38 +09:00
Namekuji
da83322200
feat: queueing bulk follow/unfollow and block/unblock (#10544)
* wrap follow/unfollow and block/unblock as job queue

* create import job to follow in each iteration

* make relationship jobs concurrent

* replace to job queue if called repeatedly

* use addBulk to import

* omit stream when importing

* fix job caller

* use ThinUser instead of User to reduce redis memory consumption

* createImportFollowingToDbJobの呼び出し方を変える, 型補強

* Force ThinUser

* オブジェクト操作のみのメソッド名はgenerate...Data

* Force ThinUser in generateRelationshipJobData

* silent bulk unfollow at admin api endpoint

---------

Co-authored-by: tamaina <tamaina@hotmail.co.jp>
2023-04-12 09:13:58 +09:00
syuilo
73203a3d72 perf(backend): cache local custom emojis 2023-04-06 11:14:43 +09:00
totegamma
9629929977
fix misskey-dev/misskey-#10417 (#10418) 2023-04-06 08:37:18 +09:00
xtexChooser
62efe2b4d5
fix: webhook delivery content-type (#9823) (#10458)
* fix: webhook delivery content-type (#9823)

* style: indent

* Update packages/backend/src/queue/processors/WebhookDeliverProcessorService.ts

Co-authored-by: Acid Chicken (硫酸鶏) <root@acid-chicken.com>

* style: trailing comma

---------

Co-authored-by: syuilo <Syuilotan@yahoo.co.jp>
Co-authored-by: Acid Chicken (硫酸鶏) <root@acid-chicken.com>
2023-04-05 14:49:58 +09:00
syuilo
f44504097c enhance(backend): improve cache 2023-04-05 10:21:10 +09:00
syuilo
ecaf152b4a enhance(backend): improve cache 2023-04-04 17:32:09 +09:00
syuilo
7f3afac0a2 refactor(backend): rename Cache -> MemoryCache 2023-04-04 15:56:47 +09:00
syuilo
30d6992684 perf(backend): 通知をRedisに保存するように
Resolve #10168
2023-04-04 14:06:57 +09:00
syuilo
b53d6c7f8c perf(backend): store notes of an antenna to redis instead of postgresql
Resolve #10169
2023-04-03 12:11:16 +09:00
syuilo
31f3f5f0f0 enhance(backend): リモートサーバーのチャート生成を無効にするオプションを追加 2023-03-24 19:08:08 +09:00
syuilo
e438091113 refactor(backend): rename cache class 2023-03-24 16:43:42 +09:00
syuilo
54630edb0f enhance: 使われてないアンテナは自動停止されるように
Resolve #9373
2023-03-20 20:12:38 +09:00
syuilo
8ae9d2eaa8 enhance: カスタム絵文字にライセンス情報を付与できるように
Resolve #10091
2023-03-16 15:08:48 +09:00
syuilo
89e2c302dd refactor(backend): integrate CreateNotificationService to NotificationService 2023-03-16 14:24:11 +09:00
syuilo
a8bd3e8e53 Update AggregateRetentionProcessorService.ts 2023-03-15 17:45:59 +09:00