Omon “physical reads” graphs indicated a clear peek of physical reads.

24 hours graph:

A few weeks graph:

A few weeks graph aggregated into 200 bars:

AWR report indicates outstanding SQL:

The sql itself is easy to tune by forcing index only acccess (not table reads) and a better index.

Patch which might help is 11865420. However if PSU’s are installed on top of 11.2 then patch merge request must be issued to Oracle.

The IGNORE_ROW_ON_DUPKEY_INDEX hint stands on itself – it changes execution semantics. Looks like it is a last minute change in Oracle 11.

All it needs is a running Oracle HTTP server in any form (as long as it supports mod_plsql) and 10g database to hold OMON PL/SQL stored procedures and few tables.

Installation zip file is under Google docs:

https://docs.google.com/open?id=0BwJA-n3_efqsWjZJejhRblBJdUE

( it wasn’t tested on RAC. I think it will aggregate statistics over all instances… )

TOP Events graph screenshot

TOP Events screenshot

Events graph screenshot:

OMON events screenshot

Statistics screenshot:

Statistics screenshot

Add database page:

Add database page screenshot

Upcomming:

Peek search module. Peek search module searches all statistics for peeks in a given interval:

Statistics peek search page

In the plans:

* drill-down possibility in graph pages. For example, pressing on “db block reads” graph column will open SQL statements ordered by db block reads.

* Collecting additional data to that of Perfstat/AWR. For example, 10g does not snapshot v$events_histogram. Snapshots of particular session level statistics like IO.  Snapshots of v$plan_statistics.  OMON repository was in the first versions of OMON but for popular use the repository must be “reimagined”.

Normal case:


select /*+full(a)*/ count(1) from iftp_dw.ar as of timestamp to_date('2012.10.05 01:00', 'yyyy.mm.dd hh24:mi:ss') a;

COUNT(1)
----------
2014667

Elapsed: 00:00:10.54

Execution Plan
----------------------------------------------------------
Plan hash value: 1818353562

-------------------------------------------------------------------
| Id | Operation | Name | Rows | Cost (%CPU)| Time |
-------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 48151 (1)| 00:09:38 |
| 1 | SORT AGGREGATE | | 1 | | |
| 2 | TABLE ACCESS FULL| AR | 1993K| 48151 (1)| 00:09:38 |
-------------------------------------------------------------------

Statistics
----------------------------------------------------------
3 recursive calls
0 db block gets
793951 consistent gets
87183 physical reads
1116 redo size
529 bytes sent via SQL*Net to client
524 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
1 rows processed


Bad case: after 3hours still running:
select /*+full(a)*/ count(1) from iftp_dw.ar as of timestamp to_date('2012 .10.05 00:00', 'yyyy.mm.dd hh24:mi:ss') a;

V$SQL statistics:

disk reads: 20365019

buffer gets: 21754220

What happened arround 00:00 A.M (AWR stats):

Every time the excessive UNDO IO is generated the process A “select from table full scan” starts at the same time as other processes which modify the same table. Those OTHER processes finish quickly. But process A then is left to slowly crawl through 400MB table for hours.

The following statistics are excessive:

-”transaction tables consistent reads – undo records applied” = 37.711.804

- “consistent changes” = 37.712.584

- “physical reads” = 36.708.796

AWR segment statistics of the Table segment (during the researched period of 7 hours):

- “db block changes” = 2.200.592. That corresponds to row count of the table.

Table segment size is: 48.128 blocks (394.264.576 bytes)

Ratio of “Excessive Statistics”/”Table Segment Size in blocks” ~ 37.000.000/48.000 = 770

I.e. to get one block of table segment about 800 blocks of physical IO must be done.

But ratio of “Excessive Statistics” / Db Block Changes(and table rows) ~ 16.

But the number of commits (AWR stats) is  2000.

Process A  is:


open cursor (full table scan);
loop
bulk fetch 1000 records into varray
for all in varray loop
do some pl/sql only processing
end loop;
end loop;
close cursor;


AWR graphs:

New AWR view columns:

New V$ view columns:

Oracle proposed resolution: “An upgrade in veritas library from 5.0 to 5.1 (which includes ODM), resolved the LGWR issue.”

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=BULLETIN&id=1320966.1

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=PROBLEM&id=790205.1

UNABLE TO START INSTANCE WITH LARGE SGA ORA-27102 SVR4 ERROR: 22: INVALID ARGUMENT [ID 790205.1]\

https://support.oracle.com/epmos/faces/ui/km/DocumentDisplay.jspx?_afrLoop=1644977629090275&id=460424.1&_afrWindowMode=0&_adf.ctrl-state=cw8egzrhl_391

Solaris 10 memory management conflicts with Automatic PGA Memory Management [ID 460424.1]

Citation (this is what I do):
“…Sleep 5 minutes between the shutdown abort and startup (this doesn’t always work) …”

Workaround: “…Essentially the /etc/system settings disable Large Page Out Of Box (LPOOB) support which was enabled by default with Solaris 10…”

One more:
“Solaris 10 memory management conflicts with Automatic PGA Memory Management [ID 460424.1]“