[WEB-2001]feat: Cache issues on the client (#5327)

* use common getIssues from issue service instead of multiple different services for modules and cycles

* Use SQLite to store issues locally and load issues from it.

* Fix incorrect total count and filtering on assignees.

* enable parallel API calls

* use common getIssues from issue service instead of multiple different services for modules and cycles

* Use SQLite to store issues locally and load issues from it.

* Fix incorrect total count and filtering on assignees.

* enable parallel API calls

* chore: deleted issue list

* - Handle local mutations
- Implement getting the updates
- Use SWR to update/sync data

* Wait for sync to complete in get issues

* Fix build errors

* Fix build issue

* - Sync updates to local-db
- Fallback to server when the local data is loading
- Wait when the updates are being fetched

* Add issues in batches

* Disable skeleton loaders for first 10 issues

* Load issues in bulk

* working version of sql lite with grouped issues

* Use window queries for group by

* - Fix sort by date fields
- Fix the total count

* - Fix grouping by created by
- Fix order by and limit

* fix pagination

* Fix sorting on issue priority

* - Add secondary sort order
- Fix group by priority

* chore: added timestamp filter for deleted issues

* - Extract local DB into its own class
- Implement sorting by label names

* Implement subgroup by

* sub group by changes

* Refactor query constructor

* Insert or update issues instead of directly adding them.

* Segregated queries. Not working though!!

* - Get filtered issues and then group them.
- Cleanup code.
- Implement order by labels.

* Fix build issues

* Remove debuggers

* remove loaders while changing sorting or applying filters

* fix loader while clearing all filters

* Fix issue with project being synced twice

* Improve project sync

* Optimize the queries

* Make create dummy data more realistic

* dev: added total pages in the global paginator

* chore: updated total_paged count

* chore: added state_group in the issues pagination

* chore: removed deleted_at from the issue pagination payload

* chore: replaced state_group with state__group

* Integrate new getIssues API, and fix sync issues bug.

* Fix issue with SWR running twice in workspace wrapper

* Fix DB initialization called when opening project for the first time.

* Add all the tables required for sorting

* Exclude description from getIssues

* Add getIssue function.

* Add only selected fields to get query.

* Fix the count query

* Minor query optimization when no joins are required.

* fetch issue description from local db

* clear local db on signout

* Correct dummy data creation

* Fix sort by assignee

* sync to local changes

* chore: added archived issues in the deleted endpoint

* Sync deletes to local db.

* - Add missing indexes for tables used in sorting in spreadsheet layout.
- Add options table

* Make fallback optional in getOption

* Kanban column virtualization

* persist project sync readiness to sqlite and use that as the source of truth for the project issues to be ready

* fix build errors

* Fix calendar view

* fetch slimed down version of modules in project wrapper

* fetch toned down modules and then fetch complete modules

* Fix multi value order by in spread sheet layout

* Fix sort by

* Fix the query when ordering by multi field names

* Remove unused import

* Fix sort by multi value fields

* Format queries and fix order by

* fix order by for multi issue

* fix loaders for spreadsheet

* Fallback to manual order whn moving away from spreadsheet layout

* fix minor bug

* Move fix for order_by when switching from spreadsheet layout to translateQueryParams

* fix default rendering of kanban groups

* Fix none priority being saved as null

* Remove debugger statement

* Fix issue load

* chore: updated isue paginated query from  to

* Fix sub issues and start and target date filters

* Fix active and backlog filter

* Add default order by

* Update the Query param to match with backend.

* local sqlite db versioning

* When window is hidden, do not perform any db versioning

* fix error handling and fall back to server when database errors out

* Add ability to disable local db cache

* remove db version check from getIssues function

* change db version to number and remove workspaceInitPromise in storage.sqlite

* - Sync the entire workspace in the background
- Add get sub issue method with distribution

* Make changes to get issues for sync to match backend.

* chore: handled workspace and project in v2 paginted issues

* disable issue description and title until fetched from server

* sync issues post bulk operations

* fix server error

* fix front end build

* Remove full workspace sync

* - Remove the toast message on sync.
- Update the disable local message.

* Add Hardcoded constant to disable the local db caching

* fix lint errors

* Fix order by in grouping

* update yarn lock

* fix build

* fix plane-web imports

* address review comments

---------

Co-authored-by: rahulramesha <rahulramesham@gmail.com>
Co-authored-by: NarayanBavisetti <narayan3119@gmail.com>
Co-authored-by: gurusainath <gurusainath007@gmail.com>
This commit is contained in:
Satish Gandham 2024-09-24 19:01:34 +05:30 committed by GitHub
parent 8dabe839f3
commit 3df230393a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
48 changed files with 2085 additions and 155 deletions

View file

@ -106,7 +106,7 @@ export class ArchivedIssues extends BaseIssuesStore implements IArchivedIssues {
});
// after fetching issues, call the base method to process the response further
this.onfetchIssues(response, options, workspaceSlug, projectId);
this.onfetchIssues(response, options, workspaceSlug, projectId, undefined, !isExistingPaginationOptions);
return response;
} catch (error) {
// set loader to undefined if errored out

View file

@ -1,6 +1,4 @@
import isArray from "lodash/isArray";
import isEmpty from "lodash/isEmpty";
import pickBy from "lodash/pickBy";
import set from "lodash/set";
import { action, computed, makeObservable, observable, runInAction } from "mobx";
// base class
@ -191,12 +189,10 @@ export class CycleIssuesFilter extends IssueFilterHelperStore implements ICycleI
});
});
const appliedFilters = _filters.filters || {};
const filteredFilters = pickBy(appliedFilters, (value) => value && isArray(value) && value.length > 0);
this.rootIssueStore.cycleIssues.fetchIssuesWithExistingPagination(
workspaceSlug,
projectId,
isEmpty(filteredFilters) ? "init-loader" : "mutation",
"mutation",
cycleId
);
await this.issueFilterService.patchCycleIssueFilters(workspaceSlug, projectId, cycleId, {
@ -237,6 +233,10 @@ export class CycleIssuesFilter extends IssueFilterHelperStore implements ICycleI
});
});
if (this.getShouldClearIssues(updatedDisplayFilters)) {
this.rootIssueStore.cycleIssues.clear(true, true); // clear issues for local store when some filters like layout changes
}
if (this.getShouldReFetchIssues(updatedDisplayFilters)) {
this.rootIssueStore.cycleIssues.fetchIssuesWithExistingPagination(
workspaceSlug,

View file

@ -179,8 +179,9 @@ export class CycleIssues extends BaseIssuesStore implements ICycleIssues {
// set loader and clear store
runInAction(() => {
this.setLoader(loadType);
this.clear(!isExistingPaginationOptions, false); // clear while fetching from server.
if (!this.groupBy) this.clear(!isExistingPaginationOptions, true); // clear while using local to have the no load effect.
});
this.clear(!isExistingPaginationOptions);
// get params from pagination options
const params = this.issueFilterStore?.getFilterParams(options, cycleId, undefined, undefined, undefined);
@ -190,7 +191,7 @@ export class CycleIssues extends BaseIssuesStore implements ICycleIssues {
});
// after fetching issues, call the base method to process the response further
this.onfetchIssues(response, options, workspaceSlug, projectId, cycleId);
this.onfetchIssues(response, options, workspaceSlug, projectId, cycleId, !isExistingPaginationOptions);
return response;
} catch (error) {
// set loader to undefined once errored out
@ -233,7 +234,7 @@ export class CycleIssues extends BaseIssuesStore implements ICycleIssues {
subGroupId
);
// call the fetch issues API with the params for next page in issues
const response = await this.issueService.getIssues(workspaceSlug, projectId, cycleId, params);
const response = await this.issueService.getIssues(workspaceSlug, projectId, params);
// after the next page of issues are fetched, call the base method to process the response
this.onfetchNexIssues(response, groupId, subGroupId);

View file

@ -103,7 +103,7 @@ export class DraftIssues extends BaseIssuesStore implements IDraftIssues {
});
// after fetching issues, call the base method to process the response further
this.onfetchIssues(response, options, workspaceSlug, projectId);
this.onfetchIssues(response, options, workspaceSlug, projectId, undefined, !isExistingPaginationOptions);
return response;
} catch (error) {
// set loader to undefined if errored out

View file

@ -65,6 +65,7 @@ export interface IBaseIssuesStore {
//actions
removeIssue: (workspaceSlug: string, projectId: string, issueId: string) => Promise<void>;
clear(shouldClearPaginationOptions?: boolean, clearForLocal?: boolean): void;
// helper methods
getIssueIds: (groupId?: string, subGroupId?: string) => string[] | undefined;
issuesSortWithOrderBy(issueIds: string[], key: Partial<TIssueOrderByOptions>): string[];
@ -455,7 +456,8 @@ export abstract class BaseIssuesStore implements IBaseIssuesStore {
options: IssuePaginationOptions,
workspaceSlug: string,
projectId?: string,
id?: string
id?: string,
shouldClearPaginationOptions = true
) {
// Process the Issue Response to get the following data from it
const { issueList, groupedIssues, groupedIssueCount } = this.processIssueResponse(issuesResponse);
@ -465,6 +467,7 @@ export abstract class BaseIssuesStore implements IBaseIssuesStore {
// Update all the GroupIds to this Store's groupedIssueIds and update Individual group issue counts
runInAction(() => {
this.clear(shouldClearPaginationOptions, true);
this.updateGroupedIssueIds(groupedIssues, groupedIssueCount);
this.loader[getGroupKey()] = undefined;
});
@ -1139,17 +1142,22 @@ export abstract class BaseIssuesStore implements IBaseIssuesStore {
/**
* Method called to clear out the current store
*/
clear(shouldClearPaginationOptions = true) {
runInAction(() => {
this.groupedIssueIds = undefined;
this.issuePaginationData = {};
this.groupedIssueCount = {};
if (shouldClearPaginationOptions) {
this.paginationOptions = undefined;
}
});
this.controller.abort();
this.controller = new AbortController();
clear(shouldClearPaginationOptions = true, clearForLocal = false) {
if (
(this.rootIssueStore.rootStore.user?.localDBEnabled && clearForLocal) ||
(!this.rootIssueStore.rootStore.user?.localDBEnabled && !clearForLocal)
) {
runInAction(() => {
this.groupedIssueIds = undefined;
this.issuePaginationData = {};
this.groupedIssueCount = {};
if (shouldClearPaginationOptions) {
this.paginationOptions = undefined;
}
});
this.controller.abort();
this.controller = new AbortController();
}
}
/**
@ -1694,7 +1702,7 @@ export abstract class BaseIssuesStore implements IBaseIssuesStore {
}
}
return isDataIdsArray ? (order ? orderBy(dataValues, undefined, [order])[0] : dataValues) : dataValues[0];
return isDataIdsArray ? (order ? orderBy(dataValues, undefined, [order]) : dataValues) : dataValues;
}
issuesSortWithOrderBy = (issueIds: string[], key: TIssueOrderByOptions | undefined): string[] => {

View file

@ -267,6 +267,20 @@ export class IssueFilterHelperStore implements IIssueFilterHelperStore {
);
};
/**
* This Method returns true if the display properties changed requires a server side update
* @param displayFilters
* @returns
*/
getShouldClearIssues = (displayFilters: IIssueDisplayFilterOptions) => {
const NON_SERVER_DISPLAY_FILTERS = ["layout"];
const displayFilterKeys = Object.keys(displayFilters);
return NON_SERVER_DISPLAY_FILTERS.some((serverDisplayfilter: string) =>
displayFilterKeys.includes(serverDisplayfilter)
);
};
/**
* This Method is used to construct the url params along with paginated values
* @param filterParams params generated from filters

View file

@ -2,6 +2,8 @@ import { makeObservable, observable } from "mobx";
import { computedFn } from "mobx-utils";
// types
import { TIssue } from "@plane/types";
// local
import { persistence } from "@/local-db/storage.sqlite";
// services
import { IssueArchiveService, IssueDraftService, IssueService } from "@/services/issue";
// types
@ -33,12 +35,14 @@ export interface IIssueStoreActions {
export interface IIssueStore extends IIssueStoreActions {
isFetchingIssueDetails: boolean;
isLocalDBIssueDescription: boolean;
// helper methods
getIssueById: (issueId: string) => TIssue | undefined;
}
export class IssueStore implements IIssueStore {
isFetchingIssueDetails: boolean = false;
isLocalDBIssueDescription: boolean = false;
// root store
rootIssueDetailStore: IIssueDetail;
// services
@ -72,8 +76,16 @@ export class IssueStore implements IIssueStore {
let issue: TIssue | undefined;
// fetch issue from local db
issue = await persistence.getIssue(issueId);
this.isFetchingIssueDetails = true;
if (issue) {
this.addIssueToStore(issue);
this.isLocalDBIssueDescription = true;
}
if (issueType === "ARCHIVED")
issue = await this.issueArchiveService.retrieveArchivedIssue(workspaceSlug, projectId, issueId, query);
else if (issueType === "DRAFT")
@ -82,7 +94,10 @@ export class IssueStore implements IIssueStore {
if (!issue) throw new Error("Issue not found");
this.addIssueToStore(issue);
const issuePayload = this.addIssueToStore(issue);
this.isLocalDBIssueDescription = false;
this.rootIssueDetailStore.rootIssueStore.issues.addIssue([issuePayload]);
// store handlers from issue detail
// parent

View file

@ -1,6 +1,4 @@
import isArray from "lodash/isArray";
import isEmpty from "lodash/isEmpty";
import pickBy from "lodash/pickBy";
import set from "lodash/set";
import { action, computed, makeObservable, observable, runInAction } from "mobx";
// base class
@ -194,12 +192,10 @@ export class ModuleIssuesFilter extends IssueFilterHelperStore implements IModul
set(this.filters, [moduleId, "filters", _key], updatedFilters[_key as keyof IIssueFilterOptions]);
});
});
const appliedFilters = _filters.filters || {};
const filteredFilters = pickBy(appliedFilters, (value) => value && isArray(value) && value.length > 0);
this.rootIssueStore.moduleIssues.fetchIssuesWithExistingPagination(
workspaceSlug,
projectId,
isEmpty(filteredFilters) ? "init-loader" : "mutation",
"mutation",
moduleId
);
await this.issueFilterService.patchModuleIssueFilters(workspaceSlug, projectId, moduleId, {
@ -240,6 +236,10 @@ export class ModuleIssuesFilter extends IssueFilterHelperStore implements IModul
});
});
if (this.getShouldClearIssues(updatedDisplayFilters)) {
this.rootIssueStore.moduleIssues.clear(true, true); // clear issues for local store when some filters like layout changes
}
if (this.getShouldReFetchIssues(updatedDisplayFilters)) {
this.rootIssueStore.moduleIssues.fetchIssuesWithExistingPagination(
workspaceSlug,

View file

@ -136,8 +136,9 @@ export class ModuleIssues extends BaseIssuesStore implements IModuleIssues {
// set loader and clear store
runInAction(() => {
this.setLoader(loadType);
this.clear(!isExistingPaginationOptions, false); // clear while fetching from server.
if (!this.groupBy) this.clear(!isExistingPaginationOptions, true); // clear while using local to have the no load effect.
});
this.clear(!isExistingPaginationOptions);
// get params from pagination options
const params = this.issueFilterStore?.getFilterParams(options, moduleId, undefined, undefined, undefined);
@ -147,7 +148,7 @@ export class ModuleIssues extends BaseIssuesStore implements IModuleIssues {
});
// after fetching issues, call the base method to process the response further
this.onfetchIssues(response, options, workspaceSlug, projectId, moduleId);
this.onfetchIssues(response, options, workspaceSlug, projectId, moduleId, !isExistingPaginationOptions);
return response;
} catch (error) {
// set loader to undefined once errored out

View file

@ -1,6 +1,4 @@
import isArray from "lodash/isArray";
import isEmpty from "lodash/isEmpty";
import pickBy from "lodash/pickBy";
import set from "lodash/set";
import { action, computed, makeObservable, observable, runInAction } from "mobx";
// base class
@ -180,13 +178,7 @@ export class ProfileIssuesFilter extends IssueFilterHelperStore implements IProf
});
});
const appliedFilters = _filters.filters || {};
const filteredFilters = pickBy(appliedFilters, (value) => value && isArray(value) && value.length > 0);
this.rootIssueStore.profileIssues.fetchIssuesWithExistingPagination(
workspaceSlug,
userId,
isEmpty(filteredFilters) ? "init-loader" : "mutation"
);
this.rootIssueStore.profileIssues.fetchIssuesWithExistingPagination(workspaceSlug, userId, "mutation");
this.handleIssuesLocalFilters.set(EIssuesStoreType.PROFILE, type, workspaceSlug, userId, undefined, {
filters: _filters.filters,

View file

@ -140,7 +140,7 @@ export class ProfileIssues extends BaseIssuesStore implements IProfileIssues {
});
// after fetching issues, call the base method to process the response further
this.onfetchIssues(response, options, workspaceSlug);
this.onfetchIssues(response, options, workspaceSlug, undefined, undefined, !isExistingPaginationOptions);
return response;
} catch (error) {
// set loader to undefined if errored out

View file

@ -1,6 +1,4 @@
import isArray from "lodash/isArray";
import isEmpty from "lodash/isEmpty";
import pickBy from "lodash/pickBy";
import set from "lodash/set";
import { action, computed, makeObservable, observable, runInAction } from "mobx";
// base class
@ -188,13 +186,11 @@ export class ProjectViewIssuesFilter extends IssueFilterHelperStore implements I
});
});
const appliedFilters = _filters.filters || {};
const filteredFilters = pickBy(appliedFilters, (value) => value && isArray(value) && value.length > 0);
this.rootIssueStore.projectViewIssues.fetchIssuesWithExistingPagination(
workspaceSlug,
projectId,
viewId,
isEmpty(filteredFilters) ? "init-loader" : "mutation"
"mutation"
);
break;
}
@ -231,6 +227,10 @@ export class ProjectViewIssuesFilter extends IssueFilterHelperStore implements I
});
});
if (this.getShouldClearIssues(updatedDisplayFilters)) {
this.rootIssueStore.projectIssues.clear(true, true); // clear issues for local store when some filters like layout changes
}
if (this.getShouldReFetchIssues(updatedDisplayFilters)) {
this.rootIssueStore.projectViewIssues.fetchIssuesWithExistingPagination(
workspaceSlug,

View file

@ -93,8 +93,9 @@ export class ProjectViewIssues extends BaseIssuesStore implements IProjectViewIs
// set loader and clear store
runInAction(() => {
this.setLoader(loadType);
this.clear(!isExistingPaginationOptions, false); // clear while fetching from server.
if (!this.groupBy) this.clear(!isExistingPaginationOptions, true); // clear while using local to have the no load effect.
});
this.clear(!isExistingPaginationOptions);
// get params from pagination options
const params = this.issueFilterStore?.getFilterParams(options, viewId, undefined, undefined, undefined);
@ -104,7 +105,7 @@ export class ProjectViewIssues extends BaseIssuesStore implements IProjectViewIs
});
// after fetching issues, call the base method to process the response further
this.onfetchIssues(response, options, workspaceSlug, projectId);
this.onfetchIssues(response, options, workspaceSlug, projectId, viewId, !isExistingPaginationOptions);
return response;
} catch (error) {
// set loader to undefined if errored out

View file

@ -1,6 +1,4 @@
import isArray from "lodash/isArray";
import isEmpty from "lodash/isEmpty";
import pickBy from "lodash/pickBy";
import set from "lodash/set";
import { action, computed, makeObservable, observable, runInAction } from "mobx";
// base class
@ -183,13 +181,7 @@ export class ProjectIssuesFilter extends IssueFilterHelperStore implements IProj
});
});
const appliedFilters = _filters.filters || {};
const filteredFilters = pickBy(appliedFilters, (value) => value && isArray(value) && value.length > 0);
this.rootIssueStore.projectIssues.fetchIssuesWithExistingPagination(
workspaceSlug,
projectId,
isEmpty(filteredFilters) ? "init-loader" : "mutation"
);
this.rootIssueStore.projectIssues.fetchIssuesWithExistingPagination(workspaceSlug, projectId, "mutation");
await this.issueFilterService.patchProjectIssueFilters(workspaceSlug, projectId, {
filters: _filters.filters,
});
@ -228,6 +220,10 @@ export class ProjectIssuesFilter extends IssueFilterHelperStore implements IProj
});
});
if (this.getShouldClearIssues(updatedDisplayFilters)) {
this.rootIssueStore.projectIssues.clear(true, true); // clear issues for local store when some filters like layout changes
}
if (this.getShouldReFetchIssues(updatedDisplayFilters)) {
this.rootIssueStore.projectIssues.fetchIssuesWithExistingPagination(workspaceSlug, projectId, "mutation");
}

View file

@ -101,8 +101,9 @@ export class ProjectIssues extends BaseIssuesStore implements IProjectIssues {
// set loader and clear store
runInAction(() => {
this.setLoader(loadType);
this.clear(!isExistingPaginationOptions, false); // clear while fetching from server.
if (!this.groupBy) this.clear(!isExistingPaginationOptions, true); // clear while using local to have the no load effect.
});
this.clear(!isExistingPaginationOptions);
// get params from pagination options
const params = this.issueFilterStore?.getFilterParams(options, projectId, undefined, undefined, undefined);
@ -112,7 +113,7 @@ export class ProjectIssues extends BaseIssuesStore implements IProjectIssues {
});
// after fetching issues, call the base method to process the response further
this.onfetchIssues(response, options, workspaceSlug, projectId);
this.onfetchIssues(response, options, workspaceSlug, projectId, undefined, !isExistingPaginationOptions);
return response;
} catch (error) {
// set loader to undefined if errored out

View file

@ -109,7 +109,7 @@ export class WorkspaceIssues extends BaseIssuesStore implements IWorkspaceIssues
});
// after fetching issues, call the base method to process the response further
this.onfetchIssues(response, options, workspaceSlug);
this.onfetchIssues(response, options, workspaceSlug, undefined, undefined, !isExistingPaginationOptions);
return response;
} catch (error) {
// set loader to undefined if errored out