[Feature] Create Pagination, Filtering, and Search Utilities for All List Endpoints

[KarenZita01]

Description

As the EquipChain API grows to serve multiple endpoints that return collections of resources (users, devices, meter readings, webhook logs, etc.), a consistent pagination, filtering, and search system is needed to provide efficient data access and a predictable developer experience. This issue creates reusable utility functions and middleware patterns that can be applied across all list endpoints.

Pagination: Implement offset-based pagination with page (1-indexed) and limit (default 20, max 100) query parameters. Responses should include metadata: { data: [...], pagination: { page, limit, total, totalPages, hasNext, hasPrev } }. The pagination utilities should work with both in-memory arrays (for MVP) and database queries (future).

Filtering: Support query parameter-based filtering with a consistent syntax: ?status=active&role=admin for exact matches, ?createdAfter=ISO and ?createdBefore=ISO for date ranges, and ?sortBy=createdAt&sortOrder=desc for sorting. Filters should be whitelist-based per endpoint to prevent arbitrary data exposure.

Search: Implement full-text search on relevant string fields via a ?q=searchTerm parameter. For the MVP in-memory implementation, this performs case-insensitive substring matching. For future database-backed implementations, this would leverage SQL LIKE or full-text indexes.

The utilities should be designed as a generic applyPagination(data, queryParams, options) function that takes an array, query parameters, and configuration (allowed filters, searchable fields, default sort), and returns the paginated, filtered, sorted result with metadata.

Technical Context & Impact

• Dependencies: No new external dependencies. Built on existing Node.js features. Will integrate with Zod schemas (Issue [Security] Implement Input Validation with Zod Schemas and Request Sanitization #8) for validating query parameters.
• Architecture: New src/utils/pagination.js with the core functions. Functions are pure and stateless, making them easy to test. Each list endpoint wraps its data source with these utilities.
• Impact: Without this feature, each list endpoint would implement its own pagination logic, leading to inconsistencies and code duplication. This utility ensures a uniform developer and API consumer experience across all endpoints.

Step-by-Step Implementation Guide

1. Create Pagination Utility: Write src/utils/pagination.js exporting paginate(data, { page, limit }) that returns { data, pagination: { page, limit, total, totalPages, hasNext, hasPrev } }. Include input validation (page >= 1, limit between 1 and 100).
2. Create Filtering Utility: In the same file, export filterData(data, filters, allowedFields) that applies exact-match filters from query params. Export applySorting(data, sortBy, sortOrder, allowedFields) that sorts by the specified field and direction.
3. Create Search Utility: Export searchData(data, query, searchableFields) that performs case-insensitive substring matching across specified fields. For MVP, use String.includes().
4. Create Combined Function: Export paginateAndFilter(data, queryParams, options) that chains search → filter → sort → paginate in a single call. Accept options.allowedFilters, options.searchableFields, options.defaultSort, options.maxLimit.
5. Create Validation Schema: Write a Zod schema paginationQuerySchema in src/schemas/common.schema.js (or update existing from Issue [Security] Implement Input Validation with Zod Schemas and Request Sanitization #8) that validates page, limit, sortBy, sortOrder, q, and any domain-specific filter params.
6. Apply to List Endpoints: Refactor existing list endpoints (admin users, devices, webhooks, etc.) to use the new utilities. Ensure all return the standardized { data, pagination } response format.
7. Write Tests: Create tests/unit/pagination.test.js with comprehensive tests: empty data, single page, multiple pages, last partial page, filtering by multiple fields, sorting ascending/descending, search with multiple terms, edge cases (page=0, negative limit, special characters in search).

Verification & Testing Steps

1. Run npm test and verify all pagination unit tests pass, including edge cases.
2. Start the server and call a list endpoint without pagination params — expect default page=1, limit=20 with the full list and pagination metadata.
3. Call with ?page=1&limit=5 — expect first 5 items with hasNext: true, total: N, totalPages: ceil(N/5).
4. Call with ?page=999 — expect an empty data array with correct pagination metadata reflecting the last valid page.
5. Call with ?q=searchTerm on a searchable endpoint — expect only items whose searchable fields contain the term (case-insensitive).
6. Call with invalid parameters (?limit=101 or ?page=-1) — expect a 400 validation error from the Zod schema.